DescriptorChain::is_valid() wrongly used .checked_offset() to attempt to
validate that the descriptor's data is in valid memory. This works in
all cases except where the guest has placed the data at the very end of
the guest memory as the offset + offset will be outside the range (as
the combined offset will be the next byte and as such out of the guest
memory). Instead use the function .check_range() takes an offset and a
length to validate
This fixes issues see with error messages featuring the
DescriptorChainTooShort error.
Fixes: #2424
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
(cherry picked from commit 1eb4ebe3d1)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
There are two parts:
- Unconditionally zero the output area. The length of the incoming
vector has been seen from 1 to 4 bytes, even though just the first
byte might need to be handled. But also, this ensures any possibly
unhandled offset will return zeroed result to the caller. The former
implementation used an I/O port which seems to behave differently from
MMIO and wouldn't require explicit output zeroing.
- An access with zero offset still takes place and needs to be handled.
Fixes#2437.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
(cherry picked from commit 9e9aba7c0b)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The following is from the Hyper-V specification v6.0b.
Cpuid leaf 0x40000003 EDX:
Bit 3: Support for physical CPU dynamic partitioning events is
available.
When Windows determines to be running under a hypervisor, it will
require this cpuid bit to be set to support dynamic CPU operations.
Cpuid leaf 0x40000004 EAX:
Bit 5: Recommend using relaxed timing for this partition. If
used, the VM should disable any watchdog timeouts that
rely on the timely delivery of external interrupts.
This bit has been figured out as required after seeing guest BSOD
when CPU hotplug bit is enabled. Race conditions seem to arise after a
hotplug operation, when a system watchdog has expired.
Closes#1799.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
(cherry picked from commit 5b168f54a6)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
error: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
Error: --> devices/src/legacy/rtc_pl031.rs:73:1
|
73 | impl Into<libc::clockid_t> for ClockType {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= note: `-D clippy::from-over-into` implied by `-D warnings`
= help: consider to implement `From` instead
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
error: redundant slicing of the whole range
Error: --> devices/src/legacy/gpio_pl061.rs:298:37
|
298 | let value = read_le_u32(&data[..]);
| ^^^^^^^^^ help: use the original slice instead: `data`
|
= note: `-D clippy::redundant-slicing` implied by `-D warnings`
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#redundant_slicing
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
error: name `GPIOInterruptDisabled` contains a capitalized acronym
Error: --> devices/src/legacy/gpio_pl061.rs:46:5
|
46 | GPIOInterruptDisabled,
| ^^^^^^^^^^^^^^^^^^^^^ help: consider making the acronym lowercase, except the initial letter: `GpioInterruptDisabled`
|
= note: `-D clippy::upper-case-acronyms` implied by `-D warnings`
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#upper_case_acronyms
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
warning: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
--> hypervisor/src/vm.rs:41:1
|
41 | impl Into<u64> for DataMatch {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= note: `#[warn(clippy::from_over_into)]` on by default
= help: consider to implement `From` instead
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into
warning: 1 warning emitted
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
warning: name `AcpiPMTimerDevice` contains a capitalized acronym
--> devices/src/acpi.rs:175:12
|
175 | pub struct AcpiPMTimerDevice {
| ^^^^^^^^^^^^^^^^^ help: consider making the acronym lowercase, except the initial letter: `AcpiPmTimerDevice`
|
= note: `#[warn(clippy::upper_case_acronyms)]` on by default
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#upper_case_acronyms
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
warning: name `IORegion` contains a capitalized acronym
--> pci/src/configuration.rs:320:5
|
320 | IORegion = 0x01,
| ^^^^^^^^ help: consider making the acronym lowercase, except the initial letter (notice the capitalization): `IoRegion`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#upper_case_acronyms
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
warning: name `ConvertFromUTF8` contains a capitalized acronym
--> virtio-devices/src/vsock/unix/mod.rs:32:5
|
32 | ConvertFromUTF8(std::str::Utf8Error),
| ^^^^^^^^^^^^^^^ help: consider making the acronym lowercase, except the initial letter: `ConvertFromUtf8`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#upper_case_acronyms
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
warning: name `InvalidCPUCount` contains a capitalized acronym
--> src/bin/ch-remote.rs:24:5
|
24 | InvalidCPUCount(std::num::ParseIntError),
| ^^^^^^^^^^^^^^^ help: consider making the acronym lowercase, except the initial letter: `InvalidCpuCount`
|
= note: `#[warn(clippy::upper_case_acronyms)]` on by default
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#upper_case_acronyms
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
warning: name `TranslateGVA` contains a capitalized acronym
--> hypervisor/src/arch/emulator/mod.rs:51:5
|
51 | TranslateGVA(#[source] anyhow::Error),
| ^^^^^^^^^^^^ help: consider making the acronym lowercase, except the initial letter: `TranslateGva`
|
= note: `#[warn(clippy::upper_case_acronyms)]` on by default
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#upper_case_acronyms
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
error: redundant slicing of the whole range
--> vhost_user_block/src/lib.rs:396:31
|
396 | right.copy_from_slice(&data[..]);
| ^^^^^^^^^ help: use the original slice instead: `data`
|
= note: `-D clippy::redundant-slicing` implied by `-D warnings`
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#redundant_slicing
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
error: name `RSDPPastRamEnd` contains a capitalized acronym
--> arch/src/lib.rs:59:5
|
59 | RSDPPastRamEnd,
| ^^^^^^^^^^^^^^ help: consider making the acronym lowercase, except the initial letter: `RsdpPastRamEnd`
|
= note: `-D clippy::upper-case-acronyms` implied by `-D warnings`
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#upper_case_acronyms
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
error: writing `&PathBuf` instead of `&Path` involves a new object where a slice will do.
--> block_util/src/lib.rs:68:31
|
68 | fn build_device_id(disk_path: &PathBuf) -> result::Result<String, Error> {
| ^^^^^^^^ help: change this to: `&Path`
|
= note: `-D clippy::ptr-arg` implied by `-D warnings`
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#ptr_arg
error: writing `&PathBuf` instead of `&Path` involves a new object where a slice will do.
--> block_util/src/lib.rs:83:39
|
83 | pub fn build_disk_image_id(disk_path: &PathBuf) -> Vec<u8> {
| ^^^^^^^^ help: change this to: `&Path`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#ptr_arg
error: writing `&PathBuf` instead of `&Path` involves a new object where a slice will do.
--> block_util/src/lib.rs:68:31
|
68 | fn build_device_id(disk_path: &PathBuf) -> result::Result<String, Error> {
| ^^^^^^^^ help: change this to: `&Path`
|
= note: `-D clippy::ptr-arg` implied by `-D warnings`
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#ptr_arg
error: writing `&PathBuf` instead of `&Path` involves a new object where a slice will do.
--> block_util/src/lib.rs:83:39
|
83 | pub fn build_disk_image_id(disk_path: &PathBuf) -> Vec<u8> {
| ^^^^^^^^ help: change this to: `&Path`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#ptr_arg
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
error: redundant slicing of the whole range
--> net_util/src/mac.rs:60:35
|
60 | bytes[..].copy_from_slice(&src[..]);
| ^^^^^^^^ help: use the original slice instead: `src`
|
= note: `-D clippy::redundant-slicing` implied by `-D warnings`
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#redundant_slicing
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
error: name `TYPE_UNKNOWN` contains a capitalized acronym
--> vm-virtio/src/lib.rs:48:5
|
48 | TYPE_UNKNOWN = 0xFF,
| ^^^^^^^^^^^^ help: consider making the acronym lowercase, except the initial letter: `Type_Unknown`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#upper_case_acronyms
error: name `GetDeviceID` contains a capitalized acronym
--> block_util/src/lib.rs:138:5
|
138 | GetDeviceID,
| ^^^^^^^^^^^ help: consider making the acronym lowercase, except the initial letter: `GetDeviceId`
|
= note: `-D clippy::upper-case-acronyms` implied by `-D warnings`
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#upper_case_acronyms
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
With CONFIG_PVH in stable kernels for some time we should deprecate the
use of alternative boot methods since this will lead to a much simpler
boot flow and CI process.
See: #2231
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
There are multiple reports of DescriptorChainTooShort errors and so add
some extra debugging to aid the debugging of this issue.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Drop the generic type E and use IrqRoutngEntry directly. This allows
dropping a bunch of trait bounds from code.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Their make_entry functions look the same now. Extract the logic to a
common function.
No functional change.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
This returns trues if this descriptor has another descriptor linked to
it. Not whether this descriptor chain has another one following it.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The 0.3.0 release (re)supports booting stock Ubuntu images so users
should be pointed towards that.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
There is no point in queueing an empty descriptor in the list of iovecs.
Let's simply ignore such case and avoid some unnecessary processing.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The hotplug documentation was missing the explanation on how to use
memory hotplug with virtio-mem.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Document that seccomp is on and how to disable it along with a pointer
on how to identify missing syscalls during development.
Fixes#993
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
It must be specified as excluded from the workspace as it must not be
built on non-test targets due to issues with the ssh2 dependency and the
musl toolchain.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
refactor vec_with_array_field to common hypervisor code
so that mshv can also make use of it.
Signed-off-by: Vineeth Pillai <viremana@linux.microsoft.com>
Bumps iced-x86 from 1.10.3 to 1.11.0.
Manual update of the code was needed since memory_displacement() was
deprecated and replaced with either memory_displacement32() or
memory_displacement64().
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
There's no need to have the code creating the passthrough_device being
duplicated since we can factorize it in a function used in both cases
(both cold plugged and hot plugged devices VFIO devices).
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Extend and use the existing DeviceTree to retrieve useful information
related to PCI devices. This removes the duplication with pci_devices
field which was internal to the DeviceManager.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Make the code a bit clearer by changing the naming of the structure
holding the list of IRQs reserved for PCI devices. It is also modified
into an array of 32 entries since we know this is the amount of PCI
slots that is supported.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We define a new enum in order to classify PCI device under virtio or
VFIO. This is a cleaner approach than using the Any trait, and
downcasting it to find the object back.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Introduces a tuple holding both information needed by pci_id_list and
pci_devices.
Changes pci_devices to be a BTreeMap of this new tuple.
Now that pci_devices holds the information needed from pci_id_list,
pci_id_list is no longer needed.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In anticipation for further factorization, the pci_id_list is now a
hashmap of PCI b/d/f leading to each device name.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Instead of relying on a PCI specific device list, we use the DeviceTree
as a reference to determine if a device name is already in use or not.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Remove the startup message containing incomplete VM configuration
details. It's a bit unusual for a tool such as Cloud Hypervisor to print
those kind of details which are a direct representation of the command
line.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This includes:
* OS disk image management
* Cloud init creation
* SSH to guest access
* Waiting for guest to boot
This will be useful in other projects that want to do similar things in
their integration tests.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Only if we have a valid API server path then create the API server. For
now this has no functional change there is a default API server path in
the clap handling but rather prepares to do so optionally.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In case of the virtio frontend driver doesn't need interupts for
certain queue event, it may explicitly write VIRTIO_MSI_NO_VECTOR
to the virtio common configuration, or it may doesn't configure
the event type vector at all.
This patch initializes both MSI-X configuration vector and queue vector
with VIRTIO_MSI_NO_VECTOR, so that the backend drivers won't trigger
unexpected interrupts to the guest.
Signed-off-by: Zide Chen <zide.chen@intel.com>
Relying on a NVIDIA Tesla T4 card present in the SGX machine, this patch
enables baremetal VFIO testing, validated by running several NVIDIA
tools in the guest. The guest image has been prepared to include all the
software needed to run these tests.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit implements ARM PrimeCell General Purpose Input/Output
(GPIO) PL061 device specification.
Signed-off-by: Penny Zheng <penny.zheng@arm.com>
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This commit switches the default serial device from 16550 to the
Arm dedicated UART controller PL011. The `ttyAMA0` can be enabled.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Enabled all "ttyS0" related test cases:
- test_serial_off
- test_serial_tty
- test_serial_file
Enabled mandatory guest kernel driver for "ns16550a" on AArch64.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
On AArch64, interrupt controller (GIC) is emulated by KVM. VMM need to
set IRQ routing for devices, including legacy ones.
Before this commit, IRQ routing was only set for MSI. Legacy routing
entries of type KVM_IRQ_ROUTING_IRQCHIP were missing. That is way legacy
devices (like serial device ttyS0) does not work.
The setting of X86 IRQ routing entries are not impacted.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Extend the existing url_to_path() to take the URL string and then use
that to simplify the snapshot/restore code paths.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Now that virtio devices can be updated with add_memory_region(), there's
no need to keep update_memory() around.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Relies on the preliminary work allowing virtio devices to be updated
with a single memory at a time instead of updating the entire memory at
once.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Assuming vhost-user devices support CONFIGURE_MEM_SLOTS protocol
feature, we introduce a new method to the VirtioDevice trait in order to
update one single memory at a time.
In case CONFIGURE_MEM_SLOTS is not supported by the backend (feature not
acked), we fallback onto the current way of updating the memory
mappings, that is with SET_MEM_TABLE.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to support GET_MAX_MEM_SLOTS, ADD_MEM_REG and REM_MEM_REG, the
protocol feature CONFIGURE_MEM_SLOTS must be enabled.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to support GET_MAX_MEM_SLOTS, ADD_MEM_REG and REM_MEM_REG, the
protocol feature CONFIGURE_MEM_SLOTS must be enabled.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The latest vhost version adds the support for the new commands
get_max_mem_slots(), add_mem_region() and remove_mem_region(), all
related to the new vhost-user protocol feature CONFIGURE_MEM_SLOTS.
The vhost_user_backend crate is updated accordingly in order to support
these new commands, mostly related to the capability of updating the
guest memory mappings with a finer control than set_mem_table() command.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
On x86_64 architecture, multiple syscalls were missing when shutting
down the vhost-user-net device along with the VM. This was causing the
usual crash related to seccomp filters.
This commit adds these missing syscalls to fix the issue.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
There is no need to get the vring base when resetting the vhost-user
device. This was mostly ignored, but in some cases, it was causing some
actual errors.
A reset must simply be a combination of disabling the vrings along with
the reset of the owner.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Originally, VhostUserSetup is only used by vhost-user-fs. While
vhost-user-blk and vhost-user-net have their own error messages,
we rename VhostUserSetup to VhostUserFsSetup.
Signed-off-by: Jiachen Zhang <zhangjiachen.jaycee@bytedance.com>
The MMIO structure contains the length rather than the maximum address
so it is necessary to subtract the starting address from the end address
to calculate the length.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In the absence of a way of integration testing this testing that it
compiles is reasonable compromise.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Load the sections backed from the file into their required addresses in
memory and populate the HOB with details of the memory. Using the HOB
address initialize the TDX state in the vCPUs and finalize the TDX
configuration.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add API to the hypervisor interface and implement for KVM to allow the
special TDX KVM ioctls on the VM and vCPU FDs.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When booting with TDX no kernel is supplied as the TDFV is responsible
for loading the OS. The requirement to have the kernel is still
currently enforced at the validation entry point; this change merely
changes function prototypes and stored state to use Option<> to support.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add support extracting the sections out for a TDVF file which can be
then used to load the TDVF and TD HOB data into their appropriate
locations.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add the skeleton of the "tdx" feature with a module ready inside the
arch crate to store implementation details.
TEST=cargo build --features="tdx"
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
These need to be updated together as the kvm-ioctls depends upon a
strictly newer version of kvm-bindings which requires a rebase in the CH
fork.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When a vm is created with a pty device, on reboot the pty fd (sub
only) will only be associated with the vmm through the epoll event
loop. The fd being polled will have been closed due to the vm itself
dropping the pty files (and potentially reopening the fd index to a
different item making things quite confusing) and new pty fds will be
opened but not polled on for input.
This change creates a structure to encapsulate the information about
the pty fd (main File, sub File and the path to the sub File). On
reboot, a copy of the console and serial pty structs is then passed
down to the new Vm instance which will be used instead of creating a
new pty device.
This resolves the underlying issue from #2316.
Signed-off-by: William Douglas <william.r.douglas@gmail.com>
Now that virtio-mem devices can update VFIO mappings through dedicated
handlers, let's provide them from the DeviceManager.
Important to note these handlers should either be provided to virtio-mem
devices or to the unique virtio-iommu device. This must be mutually
exclusive.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Create two functions for registering/unregistering DMA mapping handlers,
each handler being associated with a VFIO device.
Whenever the plugged_size is modified (which means triggered by the
virtio-mem driver in the guest), the virtio-mem backend is responsible
for updating the DMA mappings related to every VFIO device through the
handler previously provided.
It's important to update the map when the handler is either registered
or unregistered as well, as we don't want to miss some plugged memory
that would have been added before the VFIO device is added to the VM.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Instead of letting the VfioPciDevice take the decision on how/when to
perform the DMA mapping/unmapping, we move this to the DeviceManager
instead.
The point is to let the DeviceManager choose which guest memory regions
should be mapped or not. In particular, we don't want the virtio-mem
region to be mapped/unmapped as it will be virtio-mem device
responsibility to do so.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
When memory is resized through ACPI, a new region is added to the guest
memory. This region must also be added to the corresponding memory zone
in order to keep everything in sync.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Replace "--monitor-fd" with "--event-monitor" which can either take
"fd=<int>" or "path=<path>" which can point to e.g. a named pipe and
allow more flexibility.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Since the SGX server is down for maintenance, all builds are waiting on
the node agent to answer, causing all PRs to be blocked.
Let's disable temporarily the SGX CI until the server is back up.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
So far we've only had the need to emulate one instruction. There is no
need to use a HashMap when a simple tuple for the initial mapping will
do.
We can bring back the HashMap once more sophisticated use cases surface.
No functional change.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
This avoids code complexity down the line when we get around
implementing Windows support.
No functional change.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Moving to the latest version of the rust-vmm/vhost crate, before it gets
published on crates.io.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In particular update for the vmm-sys-util upgrade and all the other
dependent packages. This requires an updated forked version of
kvm-bindings (due to updated vfio-ioctls) but allowed the removal of our
forked version of kvm-ioctls.
The changes to the API from kvm-ioctls and vmm-sys-util required some
other minor changes to the code.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The vhost crate from rust-vmm is ready, which is why we do the switch
from the Cloud Hypervisor fork to the upstream crate.
At the same time, we rename the crate from vhost_rs to vhost.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit moves both pci and vmm code from the internal vfio-ioctls
crate to the upstream one from the rust-vmm project.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that ExternalDmaMapping is defined in vm-device, let's use it from
there.
This commit also defines the function get_host_address_range() to move
away from the vfio-ioctls dependency.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By moving the trait and its VFIO implementation out of vfio-ioctls, we
give anticipate for the move to the vfio-ioctls from rust-vmm.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This has been lagging behind on an older Rust version and we have
enough coverage from our Jenkins CI agent.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This is only use in the integration test and was erroneously included in
the main binary dependencies.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The interrupt tests were not being run as they were erroneously under a
feature guard that does not exist in arch.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This removes the dependency on "tempdir" which in turn depends on the
large rand dependency chain.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The help information displayed for our `--disk` option is incorrect and
incomplete, e.g. missing the `direct` and `poll_queue` field.
Signed-off-by: Bo Chen <chen.bo@intel.com>
According to Intel's mnemonic (which is used by iced-x86) the first
argument is destination while the second is source.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
The main idea behind this commit is to remove all the complexity
associated with TX/RX handling for virtio-net. By using writev() and
readv() syscalls, we could get rid of intermediate buffers for both
queues.
The complexity regarding the TAP registration has been simplified as
well. The RX queue is only processed when some data are ready to be
read from TAP. The event related to the RX queue getting more
descriptors only serves the purpose to register the TAP file if it's not
already.
With all these simplifications, the code is more readable but more
performant as well. We can see an improvement of 10% for a single
queue device.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Currently script does not exit early if the image/firmware not present
in the host. We should not progress further if the images are not pre
downloaded.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
This crate exposes the abililty for the VMM to set a file that events
should be written to. The event!() macro provides an interface to report
those events allowing the specification of an event source, an event
type and optional extra data. This will be written to the provided file
descriptor as JSON data.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
We don't have an easy way to figure out if a GPA points to normal memory
or device memory, but the guest's normal memory regions shouldn't
overlap with device regions. We can simply try to do a normal memory
read / write, and proceed to do device memory read / write if that
fails.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
This function can then be used by the TDX code to allocate the memory at
specific locations required for the TDVF to run from.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
OVMF would use string IO on those ports. String IO has not been
implemented, so that leads to panics.
Skip them explicitly in MSHV. Leave a long-ish comment in code to
explain the situation. We should properly implement string IO once it
becomes feasible / necessary.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Update for clippy in Rust 1.50.0:
error: Unnecessary nested match
--> vmm/src/vm.rs:419:17
|
419 | / if let vm_device::BusError::MissingAddressRange = e {
420 | | warn!("Guest MMIO write to unregistered address 0x{:x}", gpa);
421 | | }
| |_________________^
|
= note: `-D clippy::collapsible-match` implied by `-D warnings`
help: The outer pattern can be modified to include the inner pattern.
--> vmm/src/vm.rs:418:17
|
418 | Err(e) => {
| ^ Replace this binding
419 | if let vm_device::BusError::MissingAddressRange = e {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ with this pattern
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#collapsible_match
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
If the function can never return an error this is now a clippy failure:
error: this function's return value is unnecessarily wrapped by `Result`
--> virtio-devices/src/watchdog.rs:215:5
|
215 | / fn set_state(&mut self, state: &WatchdogState) -> io::Result<()> {
216 | | self.common.avail_features = state.avail_features;
217 | | self.common.acked_features = state.acked_features;
218 | | // When restoring enable the watchdog if it was previously enabled. We reset the timer
... |
223 | | Ok(())
224 | | }
| |_____^
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#unnecessary_wraps
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This addresses a newly added clippy issue:
error: methods called `as_*` usually take self by reference or self by mutable reference; consider choosing a less ambiguous name
--> vm-migration/src/protocol.rs:57:34
|
57 | fn as_mut_bytes<T: Sized>(p: &mut T) -> &mut [u8] {
| ^^^^^^
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#wrong_self_convention
error: aborting due to 2 previous errors
error: could not compile `vm-migration`
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Depending on the host OS the code for looking up the time for the CMOS
make require extra syscalls to be permitted for the vCPU thread.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In case the VFIO device does not support MSI or MSI-X, the capabilities
should not be parsed, avoiding the exposure of unsupported capabilities.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Make sure to propagate the error coming from VfioDevice when trying to
enable INTx, MSI or MSI-X interrutps.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Since INTx is now supported for VFIO PCI devices, there is no more
limitations regarding VFIO, which means the documentation must be
updated accordingly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
With all the preliminary work done in the previous commits, we can
update the VFIO implementation to support INTx along with MSI and MSI-X.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Here we are adding the PCI routing table, commonly called _PRT, to the
ACPI DSDT. For simplification reasons, we chose not to implement PCI
links as this involves dynamic decision from the guest OS, which result
in lots of complexity both from an AML perspective and from a device
manager perspective.
That's why the _PRT creates a static list of 32 entries, each assigned
with the IRQ number previously reserved by the device manager.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to support INTx for PCI devices, each PCI device must be
assigned an IRQ. This is preliminary work to reserve 8 IRQs which will
be shared across the 32 PCI devices.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In anticipation for accessing the legacy interrupt manager from the
function creating a VFIO PCI device, we store it as part of the
DeviceManager, to make it available for all methods.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The DeviceManager already has a hold onto the MSI interrupt manager,
therefore there's no need to pass it through every function. Instead,
let's simplify the code by using the attribute from DeviceManager's
instance.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Both GIC and IOAPIC must implement a new method notifier() in order to
provide the caller with an EventFd corresponding to the IRQ it refers
to.
This is needed in anticipation for supporting INTx with VFIO PCI
devices.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In anticipation for supporting the notifier function for the legacy
interrupt source group, we need this function to return an EventFd
instead of a reference to this same EventFd.
The reason is we can't return a reference when there's an Arc<Mutex<>>
involved in the call chain.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Swap the last two parameters of guest_mem_{read,write} to be consistent
with other read / write functions.
Use more descriptive parameter names.
No functional change.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
This reflects that it generates CPUID state used across all vCPUs.
Further ensure that errors from this function get correctly propagated.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Move the code for populating the CPUID with KVM HyperV emulation details from
the per-vCPU CPUID handling code to the shared CPUID handling code.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Move the code for populating the CPUID with details of the CPU
identification from the per-vCPU CPUID handling code to the shared CPUID
handling code.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Move the code for populating the CPUID with details of the maximum
address space from the per-vCPU CPUID handling code to the shared CPUID
handling code.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add the ability for cloud-hypervisor to create, manage and monitor a
pty for serial and/or console I/O from a user. The reasoning for
having cloud-hypervisor create the ptys is so that clients, libvirt
for example, could exit and later re-open the pty without causing I/O
issues. If the clients were responsible for creating the pty, when
they exit the main pty fd would close and cause cloud-hypervisor to
get I/O errors on writes.
Ideally the main and subordinate pty fds would be kept in the main
vmm's Vm structure. However, because the device manager owns parsing
the configuration for the serial and console devices, the information
is instead stored in new fields under the DeviceManager structure
directly.
From there hooking up the main fd is intended to look as close to
handling stdin and stdout on the tty as possible (there is some future
work ahead for perhaps moving support for the pty into the
vmm_sys_utils crate).
The main fd is used for reading user input and writing to output of
the Vm device. The subordinate fd is used to setup raw mode and it is
kept open in order to avoid I/O errors when clients open and close the
pty device.
The ability to handle multiple inputs as part of this change is
intentional. The current code allows serial and console ptys to be
created and both be used as input. There was an implementation gap
though with the queue_input_bytes needing to be modified so the pty
handlers for serial and console could access the methods on the serial
and console structures directly. Without this change only a single
input source could be processed as the console would switch based on
its input type (this is still valid for tty and isn't otherwise
modified).
Signed-off-by: William Douglas <william.r.douglas@gmail.com>
This thread is virtio-net specific, so it is not handled in the common
virtio device code.
The non-vhost implementation resumes the thread itself. Do the same
thing for vhost-user-net.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Use the newly added hugepages_size option if provided by the user to
pick a huge page size when creating the memfd region. If none is
specified use the system default.
Sadly different huge pages cannot be tested by an integration test as
creating a pool of the non-default size cannot be done at runtime
(requires kernel to be booted with certain parameters.)
TETS=Manually tested with a kernel booted with both 1GiB and 2MiB huge
pages (hugepagesz=1G hugepages=1 hugepagesz=2M hugepages=512)
Fixes: #2230
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Including a warning that the user is respsonsible for ensuring that they
have sufficient pages of the specified size.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This allows the user to use an alternative huge page size otherwise the
default size will be used.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This commit introduces a new information to the VirtioMemZone structure
in order to know if the memory zone is backed by hugepages.
Based on this new information, the virtio-mem device is now able to
determine if madvise(MADV_DONTNEED) should be performed or not. The
madvise documentation specifies that MADV_DONTNEED advice will fail if
the memory range has been allocated with some hugepages.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Signed-off-by: Hui Zhu <teawater@antfin.com>
This commit performs some refactoring to make all functions a method
from a specific object, and in particular methods for MemEpollHandler.
The point is to simplify the code to make it more readable.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Adjust the code to comply better with the virtio-mem specification by
adding some validation for the virtio-mem configuration, but also by
updating the virtio-mem configuration itself.
Nowhere in the virtio-mem specification is stated the usable region size
must be adjusted everytime the plugged size changes. For simplification
reason, and without going against the specification, the usable region
size is now kept static, setting its value to the size of the whole
region.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By introducing a ResizeSender object, we avoid having a Resize clone
with a different content than the original Resize object.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
MOV R/RM is a special case of MOVZX, so we generalize the mov_r_rm macro
to make it support both instructions.
Fixes: #2227
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The vhd module is the implementation of the VHD specification, which is
why it is important to unit test it.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Let's create a fixed VHD disk file from the existing RAW file thanks to
qemu-img, and create a new integration test to validate that
Cloud-Hypervisor can boot VHD disk image.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Relying on the simplified version of the synchronous support for RAW
disk files, the new fixed_vhd_sync module in the block_util crate
introduces the synchronous support for fixed VHD disk files.
With this patch, the fixed VHD support is complete as it is implemented
in both synchronous and asynchronous versions.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Using directly preadv and pwritev, we can simply use a RawFd instead of
a file, and we don't need to use the more complex implementation from
the qcow crate.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit adds the asynchronous support for fixed VHD disk files.
It introduces FixedVhd as a new ImageType, moving the image type
detection to the block_util crate (instead of qcow crate).
It creates a new vhd module in the block_util crate in order to handle
VHD footer, following the VHD specification.
It creates a new fixed_vhd_async module in the block_util crate to
implement the asynchronous version of fixed VHD disk file. It relies on
io_uring.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
_EJx built in should not return.
dsdt.dsl 813: Return (CEJ0 (0x00))
Warning 3104 - ^ Reserved method should not return a value (_EJ0)
dsdt.dsl 813: Return (CEJ0 (0x00))
Error 6080 - ^ Called method returns no value
Fixes: #2216
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The mutex timeout should be 0xffff rather than 0xfff to disable the
timeout feature.
dsdt.dsl 745: Acquire (\_SB.PRES.CPLK, 0x0FFF)
Warning 3130 - ^ Result is not used, possible operator timeout will be missed
dsdt.dsl 767: Acquire (\_SB.PRES.CPLK, 0x0FFF)
Warning 3130 - ^ Result is not used, possible operator timeout will be missed
dsdt.dsl 775: Acquire (\_SB.PRES.CPLK, 0x0FFF)
Warning 3130 - ^ Result is not used, possible operator timeout will be missed
Fixes: #2216
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
By using `net_util::open_tap` to create the TAP interface, the created
interface will be deleted when the returned variable (`net_utils::Tap`)
is dropped.
Signed-off-by: Bo Chen <chen.bo@intel.com>
This patch enables multi-queue support for creating virtio-net devices by
accepting multiple TAP fds, e.g. '--net fds=3:7'.
Fixes: #2164
Signed-off-by: Bo Chen <chen.bo@intel.com>
This helper can open a TAP device and configure the interface on it. If
the device needs to be opened multiple times for MQ then it also handles
that correctly.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Because of the behavior of the NVIDIA proprietary driver, we can't
expect NVIDIA cards with only MSI support to be functioning correctly
after they've been passed through with Cloud-Hypervisor.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Building with 1.51 nightly produces the following warning:
warning: unnecessary trailing semicolon
--> vmm/src/device_manager.rs:396:6
|
396 | };
| ^ help: remove this semicolon
|
= note: `#[warn(redundant_semicolons)]` on by default
warning: 1 warning emitted
Signed-off-by: Wei Liu <liuwe@microsoft.com>
The unit tests ask the Linux kernel to generate a TAP device name on
demand by passing in a format string. I suspect, but haven't been able
to confirm that there might be a rare race that triggers when creating
lots of devices in a short period of time. This is appearing in our unit
test as the occassional flake of the test_tap_read() which although it
has successfully created the device it fails to set the IP address on it
when looking it back up by it's name.
Since this is the most frequent cause of failures on our CI use a lock
to ensure that multiple TAP devices are not created simultaneously.
Fixes: #2135
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This skeleton commit brings in the support for compiling aarch64 with
the "acpi" feature ready to the ACPI enabling. It builds on the work to
move the ACPI hotplug devices from I/O ports to MMIO and conditionalises
any code that is x86_64 only (i.e. because it uses an I/O port.)
Filling in the aarch64 specific details in tables such as the MADT it
out of the scope.
See: #2178
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
e.g.
scripts/dev_cli.sh tests --integration -- --test-filter test_watchdog
This used to be supported by passing "$@" but was broken when multiple
hypervisor support was added.
Fixes: #2182
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Since QCOW and RAW synchronous implementation are very close, it makes
sense to introduce some common functions that can be shared between
these two.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
It might be useful debugging information for the user to know what kind
of disk file implementation is in use.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that BlockIoUring is the only implementation of virtio-block,
handling both synchronous and asynchronous backends based on the
AsyncIo trait, we can rename it to Block.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that both synchronous and asynchronous backends rely on the
asynchronous version of virtio-block (namely BlockIoUring), we can
get rid of the synchronous version (namely Block).
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the synchronous QCOW file implementation present in the qcow
crate, we created a new qcow_sync module in block_util that ports this
synchronous implementation to the AsyncIo trait.
The point is to reuse virtio-blk asynchronous implementation for both
synchronous and asynchronous backends.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the synchronous RAW file implementation present in the qcow
crate, we created a new raw_sync module in block_util that ports this
synchronous implementation to the AsyncIo trait.
The point is to reuse virtio-blk asynchronous implementation for both
synchronous and asynchronous backends.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the new DiskFile and AsyncIo traits, the implementation of
asynchronous block support does not have to be tied to io_uring anymore.
Instead, the only thing the virtio-blk implementation knows is that it
is using an asynchronous implementation of the underlying disk file.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Both DiskFile and AsyncIo traits are introduced to allow all kind of
files (RAW, QCOW, VHD) to be able to handle asynchronous access to the
underlying file.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Currently the GED control is in a fixed I/O port address but instead use
an MMIO address that has been chosen by the allocator.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This patch refines the sccomp filter list for the vCPU thread, as we are
no longer spawning virtio-device threads from the vCPU thread.
Fixes: #2170
Signed-off-by: Bo Chen <chen.bo@intel.com>
Even though the driver can provide fewer queues than those advertised
for some device types their is a minimum number that is required for
operation.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
It is permissable for the driver to program fewer queues than offered by
the device. Filter the queues so that only the ready ones are included
and check that they have valid addresses configured.
Fixes: #2136
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Don't assume that the number of queues provided match the number of
queues offered. The virtio spec allows the driver to program fewer
queues.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
We should use full memory barrier to ensure both guest and us
can see the correct avail_idx and avail_event_idx. Something
like this pattern:
VM: CLH:
update vring.avail->idx update avail_event_idx
mb() mb()
read avail_event_idx read vring.avail->idx
Signed-off-by: Xie Yongji <xieyongji@bytedance.com>
Rather than having to give and return the ioeventfd used for a device
clone them each time. This will make it simpler when we start handling
the driver enabling fewer queues than advertised by the device.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
We have killed the device thread by writing to the exit EventFd but we
should wait for them to quit to ensure consistency.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This will lead to the triggering of an ACPI button inside the guest in
order to cleanly shutdown the guest.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Use the ACPI GED device to trigger a notitifcation of type
POWER_BUTTON_CHANGED which will ultimately lead to the guest being
notified.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
If a GED event of type 0x8 (for power button) is received notify the
guest OS that this button device has been activated.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Renamed this bitfield as it will also be used for non-hotplug purposes
such as synthesising a power button.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In order to make the thread naming more useful derive their name from
the device id (which can be supplied by the user) and a device specific
suffix that has details of the individual queue (or queue pair.)
e.g.
rob@artemis:~$ pstree -p -c -l -t `pidof cloud-hypervisor`
cloud-hyperviso(27501)─┬─{_console}(27525)
├─{_disk0_q0}(27529)
├─{_disk0_q1}(27532)
├─{_net1_ctrl}(27533)
├─{_net1_qp0}(27534)
├─{_net1_qp1}(27535)
├─{_rng}(27526)
├─{http-server}(27504)
├─{seccomp_signal_}(27502)
├─{signal_handler}(27523)
├─{vcpu0}(27520)
├─{vcpu1}(27522)
└─{vmm}(27503)
Fixes: #2077
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The Windows image is quite large (about 20GiB), hence it takes some time
to copy it for every test in order to avoid potential corruption.
One way to mitigate that without compromising on safety between each
test is by using device mapper. By creating a read-only base, we ensure
the image won't be modified by any of the tests, and by creating one
snapshot for each test, we avoid copying the entire image each time.
A dedicated Copy On Write disk image is created to handle any change
that might be performed on the base image, letting the tests behave as
expected.
Fixes#2155
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By relying on the Guest object, Windows dedicated tests copy the Windows
guest image before booting from it. The point being to avoid corruption
between multiple tests. This is already how the rest of the integration
tests work, Windows tests were the only ones missing this feature.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The tight coupling of these packages caused issues for dependabot trying
to update them separately so instead combine them together into one
update.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This image does not have the pollinate service which can sometimes fail
and prevent SSH from starting as it marks itself as a prerequisite. This
service will never fully succeed as it tries to make a network
connection which will fail inside our test VMs.
Fixes: #2113
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This allows the removal of most of the custom patches in particular all
the virtio-fs ones have been integrated.
Fixes: #2070
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The ch-5.10.6 branch is rebased on top of Linux stable and includes the
latest virtio-iommu support patches.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Using --net=host is not necessary for any of the integration tests, so
let's use the default network option called "bridge".
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Older libc (like RHEL7) uses open() rather than openat(). This was
demonstrated through a failure to open /etc/localtime as used by
gmtime() libc call trigged from the vCPU thread (CMOS device.)
Fixes: #2111
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Because we're back on transient builder, let's download the image
everytime.
This reverts commit b5653d5278.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Some sporadic failures were due to an early connection to the VM while
it was not fully ready. Increasing sleep times fixes these issues.
Fixes#2104
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Sometimes when running under the CI tests fail due to a barrier not
being released and the guest blocks on an MMIO write. Add further
debugging to try and identify the issue.
See: #2118
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Given we already check the connected IP address matches the expected
guest IP address, the check on the "booted" message is not needed.
Fixes: #2117
Signed-off-by: Bo Chen <chen.bo@intel.com>
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
On aarch64, the openat() syscall was missing from the seccomp filters
list, preventing the test_watchdog from running properly.
Fixes#2103
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Temporarily disable fast failing to try and make progress on CI
stabilisation to allow more test coverage.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This test is very flaky and regularly causing CI failures. Until we can
identify the root cause we should disable this test.
See: #2103
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Simplify our image handling by not copying both QCOW2 and raw images for
every test. Allow the test to choose QCOW2 or raw by specifying the
image name manually. A follow on patch will add explicity QCOW2 tests.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When an SSH command fails we want to be able to see, via a panic() why
and where it failed. Replace use of .unwrap_or_default() from SSH
command calls to ensure that we can see the location of the panic.
Also enhance the existing SSH output code to show the error if there is
one.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Based on the LKML feedback, the devices under /dev/sgx/* are
not justified. SGX RFC v40 moves the SGX device nodes to /dev/sgx_*
and this is reflected in kvm-sgx (next branch) too.
Update cloud-hypervisor code and documentation to follow this.
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
This reverts commit 7f1f335950.
This breaks the compilation on Fedora/RedHat machines by requiring
dpkg-deb in the build.rs
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This allows the removal of most of the custom patches in particular all
the virtio-fs ones have been integrated.
Partially fixes: #2070
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
If the vCPU thread calls log!() the time difference between the call
time and the boot up time is reported. On most environments and
architectures this covered by a vDSO call rather than a syscall. However
on some platforms this turns into a syscall.
Fixes: #2080
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
error: field assignment outside of initializer for an instance created with Default::default()
Error: --> hypervisor/src/kvm/mod.rs:1239:9
|
1239 | state.mp_state = self.get_mp_state()?;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= note: `-D clippy::field-reassign-with-default` implied by `-D warnings`
note: consider initializing the variable with `kvm::aarch64::VcpuKvmState { mp_state: self.get_mp_state()?, ..Default::default() }` and removing relevant reassignments
--> hypervisor/src/kvm/mod.rs:1237:9
|
1237 | let mut state = CpuState::default();
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#field_reassign_with_default
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
error: field assignment outside of initializer for an instance created with Default::default()
--> virtio-devices/src/mem.rs:496:9
|
496 | resp.resp_type = resp_type;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
note: consider initializing the variable with `mem::VirtioMemResp { resp_type: resp_type, ..Default::default() }` and removing relevant reassignments
--> virtio-devices/src/mem.rs:495:9
|
495 | let mut resp = VirtioMemResp::default();
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#field_reassign_with_default
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
We must explicitly mark these values as u8 as the function that consumes
them takes a T and needs to use the specific width.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
error: field assignment outside of initializer for an instance created with Default::default()
--> hypervisor/src/kvm/mod.rs:318:9
|
318 | cap.cap = KVM_CAP_SPLIT_IRQCHIP;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= note: `-D clippy::field-reassign-with-default` implied by `-D warnings`
note: consider initializing the variable with `kvm_bindings::kvm_enable_cap { cap: KVM_CAP_SPLIT_IRQCHIP, ..Default::default() }` and removing relevant reassignments
--> hypervisor/src/kvm/mod.rs:317:9
|
317 | let mut cap: kvm_enable_cap = Default::default();
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#field_reassign_with_default
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
error: this returns a `Result<_, ()>
--> net_util/src/mac.rs:68:5
|
68 | pub fn from_bytes(src: &[u8]) -> Result<MacAddr, ()> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= note: `-D clippy::result-unit-err` implied by `-D warnings`
= help: use a custom Error type instead
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#result_unit_err
Replace with std::io::Error like other locations in the same file.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
With Rust 1.49 using attributes on a function parameter is not allowed.
The recommended workaround is to put it in a new block.
error[E0658]: attributes on expressions are experimental
--> vmm/src/memory_manager.rs:698:17
|
698 | #[cfg(target_arch = "x86_64")]
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= note: see issue #15701 <https://github.com/rust-lang/rust/issues/15701> for more information
error: removing an expression is not supported in this position
--> vmm/src/memory_manager.rs:698:17
|
698 | #[cfg(target_arch = "x86_64")]
|
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The boot time for direct kernel boot based tests is significantly
quicker than booting via the firmware and stock kernel as it triggers a
reboot during the boot process due to the initrd handling.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When doing a direct kernel boot only have console=ttyS0 in the command
line if we are explicitly testing the serial output. The default
behaviour is `--serial null` so this output will not be visible but will
trigger a KVM exit for every byte which is very costly when running
under nested virtualization.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
On the CI we are seeing issues with the activation barriers not being released:
cloud-hypervisor: 12.452434193s: INFO:vmm/src/vm.rs:413 -- Waiting for barrier
cloud-hypervisor: 12.452499794s: INFO:virtio-devices/src/block.rs:382 -- Changing cache mode to writeback
cloud-hypervisor: 12.452605195s: INFO:vmm/src/vm.rs:413 -- Waiting for barrier
cloud-hypervisor: 12.452684596s: INFO:virtio-devices/src/transport/pci_device.rs:671 -- Waiting for barrier
cloud-hypervisor: 12.452708196s: INFO:virtio-devices/src/transport/pci_device.rs:673 -- Barrier released
cloud-hypervisor: 12.452717596s: INFO:vmm/src/vm.rs:415 -- Barrier released
Add some debugging to try and identify the vause of this issue.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Now that we can support supplying an fd to the TAP device we can support
the MACVTAP and MACVLAN use cases.
Fixes: #872
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add an "fd=" parameter to allow specifying a TAP fd to use. Currently
only one fd for one queue pair is supported.
Fixes: #2052
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add support for creating virtio-net device from existing TAP fd.
Currently only a single fd and thus no-more than 2 queues (one pair) is
suppored.
Fixes: #2052
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Ultimately this will allow the creation of a virtio-net device that is
backed by a file descriptor.
This function ensures that the TAP device is correctly setup with
offloading and non-blocking.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
And fix related warnings: op_kind and op_register are being deprecated
as they might panic.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Starting the virtio device threads from the VMM thread has slowed down
the start of the VM when running on a highly contested system like the
CI.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When a device is ready to be activated signal to the VMM thread via an
EventFd that there is a device to be activated. When the VMM receives a
notification on the EventFd that there is a device to be activated
notify the device manager to attempt to activate any devices that have
not been activated.
As a side effect the VMM thread will create the virtio device threads.
Fixes: #1863
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
We need to be able to return the barrier from the code that prepares to
activate the virtio device. This triggered by a write to the
configuration fields stored in the PCI BAR. Since bars can be accessed
by both memory mapping and through PCI config I/O several prototypes
must be changed.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This can be uses to indicate to the caller that it should wait on the
barrier before returning as there is some asynchronous activity
triggered by the write which requires the KVM exit to block until it's
completed.
This is useful for having vCPU thread wait for the VMM thread to proceed
to activate the virtio devices.
See #1863
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Currently these two macros(msr, msr_data) reside both on kvm and mshv
module. Definition is same for both module. Moving them to arch/x86
module eliminates redundancy and makes more sense.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
On the CI we are seeing that sometimes the epoll is receiving these
errors which do not indicate a failure but that we should retry.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Since we can't test mshv and kvm at the same time, --all-features no
longer work.
We factorize all, non-hypervisor related features into a common set and
mix that with either mshv and kvm.
Co-Developed-by: Samuel Ortiz <sameo@linux.intel.com>
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Co-Developed-by: Wei Liu <liuwe@microsoft.com>
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Signed-off-by: Muminul Islam <muislam@microsoft.com>
We don't have IrqFd and IOEventFd support in the kernel for now.
So an emulation layer is needed. In the future, we will be adding this
support in the kernel.
Co-Developed-by: Wei Liu <liuwe@microsoft.com>
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Signed-off-by: Muminul Islam <muislam@microsoft.com>
vmmops trait object is needed to get access some
of the upper level vmm functionalities i.e guest
memory access, IO read write etc.
Co-Developed-by: Praveen Paladugu <prapal@microsoft.com>
Signed-off-by: Praveen Paladugu <prapal@microsoft.com>
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Adding hv_state (hyperv state) to Vm and Vcpu struct for mshv.
This state is needed to keep some kernel data(for now hypercall page)
in the vmm.
Co-Developed-by: Praveen Paladugu <prapal@microsoft.com>
Signed-off-by: Praveen Paladugu <prapal@microsoft.com>
Signed-off-by: Muminul Islam <muislam@microsoft.com>
This is the initial folder structure of the mshv module inside
the hypervisor crate. The aim of this module is to support Microsoft
Hyper-V as a supported Hypervisor.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
There are some code base and function which are purely KVM specific for
now and we don't have those supports in mshv at the moment but we have plan
for the future. We are doing a feature guard with KVM. For example, KVM has
mp_state, cpu clock support, which we don't have for mshv. In order to build
those code we are making the code base for KVM specific compilation.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
The customized hashmap macro can't be lifted to common MockVMM code.
MockVMM only needs a collection to iterate over to get initial register
states. A vector is just as good as a hashmap.
Switch to use a vector to store initial register states. This allows us
to drop the hashmap macro everywhere.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Unfortunately Rust stable does not yet have inline ASM support the flag
calculation will have to be implemented in software.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
nightly / beta are not required actions but if they fail then all other
builds will be cancelled and so prevent the PR from being cleared as
buildable.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Unfortunately it seems patch entries are ignored when obtaining
dependencies from another workspace.
Remove the problematic kvm-ioctls and kvm-bindings patch entries and use
the forked repository directly.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When using an PIO write to 0x80 which is a special case handle that and
then return without going through the resolve.
This removes an extra warning that is reported.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
There is no need to have a pair of curly brackets for structures without
any member.
No functional change.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
The mapping between code and its handler is static. We can drop the
HashMap in favour of a static match expression.
This has two benefits:
1. No more memory allocation and deallocation for the HashMap.
2. Shorter look-up time.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
When a total ordering between multiple atomic variables is not required
then use Ordering::Acquire with atomic loads and Ordering::Release with
atomic stores.
This will improve performance as this does not require a memory fence
on x86_64 which Ordering::SeqCst will use.
Add a comment to the code in the vCPU handling code where it operates on
multiple atomics to explain why Ordering::SeqCst is required.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The DeviceNode cannot be fully represented as it embeds a Rust style
enum (i.e. with data) which is instead represented by a simple
associative array.
Fixes: #1167
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When the x86 instruction decoder tells us about some missing bytes from
the instruction stream, we call into the platform fetch method and
emulate one last instruction.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
In preparation for the instruction fetching step, we modify the decoding
loop so that we can check what the last decoding error is.
We also switch to explictly using decode_out() which removes a 32 bytes
copy compared to decode().
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
In order to validate emulated memory accesses, we need to be able to get
all the segments descriptor attributes.
This is done by abstracting the SegmentRegister attributes through a
trait that each hypervisor will have to implement.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
We need to be able to build the CPU mode from its state in order to
start implementing mode related checks in the x86 emulator.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
If we rely on timeouts at the top level we can get builds being aborted
simply because they took too long to be scheduled rather than because
the actual integration tests took too long.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The MockVMM platform will be used by other instructions emulation
implementations, but also by the emulator framework.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The observation here is PlatformEmulator can be seen as the context for
emulation to take place. It should be rather easy to construct a context
that satisfies the lifetime constraints for instruction emulation.
The thread doing the emulation will have full ownership over the
context, so this removes the need to wrap PlatformEmulator in Arc and
Mutex, as well as the need for the context to be either Clone or Copy.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
The emulator gets a CPU state from a CpuStateManager instance, emulates
the passed instructions stream and returns the modified CPU state.
The emulator is a skeleton for now since it comes with an empty
instruction mnemonic map.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
And an InstructionMap helper structure to map x86 mnemonic codes
to instruction handlers.
Any instruction emulation implementation should then boil down with
implementing InstructionHandler for any supported mnemonic.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Minimal will be defined by the amount of emulated instructions.
Carrying all GPRs, all CRs, segment registers and table registers should
cover quite a few instructions.
Co-developed-by: Wei Liu <liuwe@microsoft.com>
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
For efficiently emulating x86 instructions, we need to build and pass a
CPU state copy/reference to instruction emulation handlers. Those handlers
will typically modify the CPU state and let the caller commit those
changes back through the PlatformEmulator trait set_cpu_state method.
Hypervisors typically have internal CPU state structures, that maps back
to the correspinding kernel APIs. By implementing the CpuState trait,
instruction emulators will be able to directly work on CPU state
instances that are directly consumable by the underlying hypervisor and
its kernel APIs.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
In order to emulate instructions, we need a way to get access to some of
the guest resources. The PlatformEmulator interface provides guest
memory and CPU state access to emulator implementations.
Typically, an hypervisor will implement PlatformEmulator for architecture
specific instruction emulators to build their framework on top of.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
We will need the GDT API for the hypervisor's x86 instruction
emulator implementation, it's better if the arch crate depends on the
hypervisor one rather than the other way around.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The configuration is stored separately to the Vm in the VMM. The failure
to store the config was preventing the VM from shutting down correctly
as Vmm::vm_delete() checks for the presence of the config.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Rather return the None to the caller to handle instead. This removes the
source of a potential panic.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The live migration support added use of this ioctl but it wasn't
included in the permitted list.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This interface is used by the vCPU thread to delegate responsibility for
handling MMIO/PIO operations and to support different approaches than a
VM exit.
During profiling I found that we were spending 13.75% of the boot CPU
uage acquiring access to the object holding the VmmOps via
ArcSwap::load_full()
13.75% 6.02% vcpu0 cloud-hypervisor [.] arc_swap::ArcSwapAny<T,S>::load_full
|
---arc_swap::ArcSwapAny<T,S>::load_full
|
--13.43%--<hypervisor::kvm::KvmVcpu as hypervisor::cpu::Vcpu>::run
std::sys_common::backtrace::__rust_begin_short_backtrace
core::ops::function::FnOnce::call_once{{vtable-shim}}
std::sys::unix::thread::Thread::new::thread_start
However since the object implementing VmmOps does not need to be mutable
and it is only used from the vCPU side we can change the ownership to
being a simple Arc<> that is passed in when calling create_vcpu().
This completely removes the above CPU usage from subsequent profiles.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
As we switched to focal for this test we no longer get any output during
the boot unless serial is used over virtio-console.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
There have been a lot of flakes around tests such as
test_virtio_fs_hotplug_dax_on_w_vhost_user_fs_daemon() or
test_virtio_fs_hotplug_dax_on() which all try and hotplug memory.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add config parameter to --disk called "_disable_io_uring" (the
underscore prefix indicating it is not for public consumpion.) Use this
option to disable io_uring if it would otherwise be used.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
With the removal of vhost-user self-spawning support we should migrate
the tests to use the binaries so that we can remove the functionality
from the cloud-hypervisor binary itself.
See: #1925
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Now the VM is paused/resumed by the migration process itself.
0. The guest configuration is sent to the destination
1. Dirty page log tracking is started by start_memory_dirty_log()
2. All guest memory is sent to the destination
3. Up to 5 attempts are made to send the dirty guest memory to the
destination...
4. ...before the VM is paused
5. One last set of dirty pages is sent to the destination
6. The guest is snapshotted and sent to the destination
7. When the migration is completed the destination unpauses the received
VM.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This allows code running in the VMM to access the VM's MemoryManager's
functionality for managing the dirty log including resetting it but also
generating a table.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Prior to sending the memory the full state is not needed only the
configuration. This is sufficient to create the appropriate structures
in the guest and have the memory allocations ready for filling.
Update the protocol documentation to add a separate config step and move
the state to after the memory is transferred. As the VM is created in a
separate step to restoring it the requires a slightly different
constructor as well as saving the VM object for the subsequent commands.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In order to do this we must extend the MemoryManager API to add the
ability to specify the tracking of the dirty pages when creating the
userspace mappings and also keep track of the userspace mappings that
have been created for RAM regions.
Currently the dirty pages are collected into ranges based on a block
level of 64 pages. The algorithm could be tweaked to create smaller
ranges but for now if any page in the block of 64 is dirty the whole
block is added to the range.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
While the addressable space size reduction of 4k in necessary due to
the Linux bug, the 64k alignment of the addressable space size is
required by Windows. This patch satisfies both.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
Cloning the ArcSwapOption (like the ArcSwap) does not act like a
.clone() on an Arc, instead an entirely new ArcSwap is created with the
same contents. To correctly share the ArcSwap needs to be placed inside
an Arc.
See: 2433d5719b (diff-6c6d94533c44c19bd1416ef17bad1a878e63dca6e98d59181228fbe8f967c62bR6)
Due to this being wrongly used ::clone() was removed from
ArcSwap/ArcSwapOption in 1.0.0.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This is tested by:
Source VMM:
target/debug/cloud-hypervisor --kernel ~/src/linux/vmlinux \
--pmem file=~/workloads/focal.raw --cpus boot=1 \
--memory size=2048M \
--cmdline"root=/dev/pmem0p1 console=ttyS0" --serial tty --console off \
--api-socket=/tmp/api1 -v
Destination VMM:
target/debug/cloud-hypervisor --api-socket=/tmp/api2 -v
And the following commands:
target/debug/ch-remote --api-socket=/tmp/api1 pause
target/debug/ch-remote --api-socket=/tmp/api2 receive-migration unix:/tmp/foo &
target/debug/ch-remote --api-socket=/tmp/api1 send-migration unix:/tmp/foo
target/debug/ch-remote --api-socket=/tmp/api2 resume
The VM is then responsive on the destination VMM.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This allows the code to be reused when creating the VM from a snapshot
when doing VM migration.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add API entry points with stub implementation for sending and receiving
a VM from one VMM to another.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Due to a known limitation in OpenAPITools/openapi-generator tool,
it's impossible to send go zero types, like false and 0 to
cloud-hypervisor because `omitempty` is added if a field is not
required.
Set cache_size, dax, num_queues and queue_size as required to remove
`omitempty` from the json tag.
fixes#1961
Signed-off-by: Julio Montes <julio.montes@intel.com>
This also removes the need to lookup up the "exe" symlink for finding
the VMM executable path.
Fixes: #1925
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The integration script failed to obtain the new guest kernel commit when
the code folder had been existing on CI machine.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
The logic to handle AArch64 system event was: SHUTDOWN and RESET were
all treated as RESET.
Now we handle them differently:
- RESET event will trigger Vmm::vm_reboot(),
- SHUTDOWN event will trigger Vmm::vm_shutdown().
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Now Vcpu::run() returns a boolean value to VcpuManager, indicating
whether the VM is going to reboot (false) or just continue (true).
Moving the handling of hypervisor VCPU run result from Vcpu to
VcpuManager gives us the flexibility to handle more scenarios like
shutting down on AArch64.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Set the test case test_snapshot_restore X86 only, instead of excluding
it from test command line.
The command line option was added because we used to support migration
with Virtio-MMIO, but not Virtio-PCI.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Rather than filling the guest memory from a file at the point of the the
guest memory region being created instead fill from the file later. This
simplifies the region creation code but also adds flexibility for
sourcing the guest memory from a source other than an on disk file.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Tests not ported include 1) the ones that start guest VMs without
network (e.g. test_net_hotplug, test_initramfs), 2) test_vfio that
involves l2 guest. Also, some tests that use bionic guest image are
given extended timeout (120s) for 'wait_vm_boot'.
Signed-off-by: Bo Chen <chen.bo@intel.com>
Instead of waiting blindly with fixed amount of sleeping time, we can
use the `wait-timeout` crate to explicitly wait VM shutdown (with a
timeout). It can reduces the execution time of some tests
substantially. Also, this patch increases the `shutdown` timeout for
'test_reboot', which should fix the recent sporadic failures on this
test.
Signed-off-by: Bo Chen <chen.bo@intel.com>
Address build failure from activity in the development virtio-fs branch
by using the stable fork.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Instead of blindly waiting for 20-40s for the guest VM to boot, this
patch waits the notification from the guest VM explicitly by using a
simple TcpListener on the host and a custom systemd service in the
guest.
This patch also ported few tests to use this new machanism, while more
tests are to be ported.
Signed-off-by: Bo Chen <chen.bo@intel.com>
As a mirror of bdbea19e23 which ensured
that GuestMemoryMmap::read_exact_from() was used to read all the file to
the region ensure that all the guest memory region is written to disk.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Use a Result<> type with an error to simplify the code in start_vmm().
This will also make it easier to add cleanup funtionality.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This gives a nicer user experience and this error can now be used as the
source for other errors based off this.
See: #1910
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Now that we that our CI is running with a kernel that is new enough to
support io_uring we can turn this feature on by default.
See: #1561
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Any occurrence of of a variable containing `ext_region` is replaced with
the less confusing name `saved_region`. The point is to clearly identify
the memory regions that might have been saved during a snapshot, while
the `ext` standing for `external` was pretty unclear.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In the context of saving the memory regions content through snapshot,
using the term "backing file" brings confusion with the actual backing
file that might back the memory mapping.
To avoid such conflicting naming, the 'backing_file' field from the
MemoryRegion structure gets replaced with 'content', as this is
designating the potential file containing the memory region data.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Use GuestRegionMmap::read_exact_from() to ensure that all of the file is
read into the guest. This addresses an issue where
GuestRegionMmap::read_from() was only copying the first 2GiB of the
memory and so lead to snapshot-restore was failing when the guest RAM
was 2GiB or greater.
This change also propagates any error from the copying upwards.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Split out the HTTP request handling code from ch-remote into a new
crate which can be used in other places where talking to the API server
by HTTP is necessary.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Use the trait for Read/Write rather than specifying the concrete type.
This allows for the functionality to be used for different socket types.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When restoring if a region of RAM is backed by anonymous memory i.e from
memfd_create() then copy the contents of the ram from the file that has
been saved to disk.
Previously the code would map the memory from that file into the guest
using a MAP_PRIVATE mapping. This has the effect of
minimising the restore time but provides an issue where the restored VM
does not have the same structure as the snapshotted VM, in particular
memory is backed by files in the restored VM that were anonymously
backed in the original.
This creates two problems:
* The snapshot data is mapped from files for the pages of the guest
which prevents the storage from being reclaimed.
* When snapshotting again the guest memory will not be correctly saved
as it will have looked like it was backed by a file so it will not be
written to disk but as it is a MAP_PRIVATE mapping the changes will
never be written to the disk again. This results in incorrect
behaviour.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Now that virtio-balloon is not declared as part of the --memory
parameter, the integration tests are updated to keep the correct
behavior.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The standalone `--balloon` parameter being fully functional at this
point, we can get rid of the balloon options from the --memory
parameter.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that we have a new dedicated way of asking for a balloon through the
CLI and the REST API, we can move all the balloon code to the device
manager. This allows us to simplify the memory manager, which is already
quite complex.
It also simplifies the behavior of the balloon resizing command. Instead
of providing the expected size for the RAM, which is complex when memory
zones are involved, it now expects the balloon size. This is a much more
straightforward behavior as it really resizes the balloon to the desired
size. Additionally to the simplication, the benefit of this approach is
that it does not need to be tied to the memory manager at all.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This introduces a new way of defining the virtio-balloon device. Instead
of going through the --memory parameter, the idea is to consider balloon
as a standalone virtio device.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The snasphot/restore feature is not working because some CPU states are
not properly saved, which means they can't be restored later on.
First thing, we ensure the CPUID is stored so that it can be properly
restored later. The code is simplified and pushed down to the hypervisor
crate.
Second thing, we identify for each vCPU if the Hyper-V SynIC device is
emulated or not. In case it is, that means some specific MSRs will be
set by the guest. These MSRs must be saved in order to properly restore
the VM.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The watchdog device is created through the "--watchdog" parameter. At
most a single watchdog can be created per VM.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This device operates a single virtq. When the driver offers a descriptor
to the device it is interpreted as a "ping" to indicate that the guest
is alive. A periodic timer fires and if when the timer is fired there
has not been a "ping" from the guest then the device will reset the VM.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Before Virtio-mmio was removed, we passed an optional PCI space address
parameter to AArch64 code for generating FDT. The address is none if the
transport is MMIO.
Now Virtio-PCI is the only option, the parameter is mandatory.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Given the increased amount of output from cloud-hypervisor, this patch
also increased the PIPE_SIZE to 32MB (from 256KB).
Signed-off-by: Bo Chen <chen.bo@intel.com>
This patch prints the complete commandline when launching
cloud-hypervisor. It also prints the details of the `ssh` command if
the command is failing.
Signed-off-by: Bo Chen <chen.bo@intel.com>
Virtio-mmio is removed, now virtio-pci is the only option for virtio
transport layer. We use MSI for PCI device interrupt. While GICv2, the
legacy interrupt controller, doesn't support MSI. So GICv2 is not very
practical for Cloud-hypervisor, we can remove it.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
When shutting down a VM using VFIO, the following error has been
detected:
vfio-ioctls/src/vfio_device.rs:312 -- Could not delete VFIO group:
KvmSetDeviceAttr(Error(9))
After some investigation, it appears the KVM device file descriptor used
for removing a VFIO group was already closed. This is coming from the
Rust sequence of Drop, from the DeviceManager all the way down to
VfioDevice.
Because the DeviceManager owns passthrough_device, which is effectively
a KVM device file descriptor, when the DeviceManager is dropped, the
passthrough_device follows, with the effect of closing the KVM device
file descriptor. Problem is, VfioDevice has not been dropped yet and it
still needs a valid KVM device file descriptor.
That's why the simple way to fix this issue coming from Rust dropping
all resources is to make Linux accountable for it by duplicating the
file descriptor. This way, even when the passthrough_device is dropped,
the KVM file descriptor is closed, but a duplicated instance is still
valid and owned by the VfioContainer.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We turn on that emulation for Windows. Windows does not have KVM's PV
clock, so calling notify_guest_clock_paused results in an error.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
While using the virtio-iommu device involving L2 scenario, and tearing
things down all the way from L2 back to L0 exposed some bad syscalls
that were not part of the authorized list.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that we've written Windows integration tests and the associated
script to launch them, this patch enables the support for Windows tests
in dev_cli.sh, so that we can run it in our Cloud Hypervisor container.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This is a new integration test running Windows as a guest with Cloud
Hypervisor. Once the VM is booted, the test connects to the guest
through SSH and shutdown the VM. If this succeeds, this means the VM
was properly booted to userspace and that the network was functional.
Important to note that because this test generates lots of logs, it
requires a large pipe size for both stdout and stderr.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Introduce a new test that will validate the new option `max_phys_bits`
from the `--cpus` parameter behaves as expected.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
If the user specified a maximum physical bits value through the
`max_phys_bits` option from `--cpus` parameter, the guest CPUID
will be patched accordingly to ensure the guest will find the
right amount of physical bits.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
If the user provided a maximum physical bits value for the vCPUs, the
memory manager will adapt the guest physical address space accordingly
so that devices are not placed further than the specified value.
It's important to note that if the number exceed what is available on
the host, the smaller number will be picked.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to let the user choose maximum address space size, this patch
introduces a new option `max_phys_bits` to the `--cpus` parameter.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The 'GuestAddress::unchecked_add' function has undefined behavior when
an overflow occurs. Its alternative 'checked_add' requires use to handle
the overflow explicitly.
Signed-off-by: Bo Chen <chen.bo@intel.com>
We are now reserving a 256M gap in the guest address space each time
when hotplugging memory with ACPI, which prevents users from hotplugging
memory to the maximum size they requested. We confirm that there is no
need to reserve this gap.
This patch removes the 'reserved gaps'. It also refactors the
'MemoryManager::start_addr' so that it is rounding-up to 128M alignment
when hotplugged memory is allowed with ACPI.
Signed-off-by: Bo Chen <chen.bo@intel.com>
In previous dev_cli.sh, the `uname -m` command will generate
either `x86_64` or `aarch64`, which is inconsistent with the
architectures in the Dockerfile, namely `amd64` and `arm64`.
This will cause some dependancy missing in the docker container
when the docker image is built locally.
This commit fixes this inconsistancy.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
We now try to create a ram region of size 0 when the requested memory
size is the same as current memory size. It results in an error of
`GuestMemoryRegion(Mmap(Os { code: 22, kind: InvalidInput, message:
"Invalid argument" }))`. This error is not meaningful to users and we
should not report it.
Signed-off-by: Bo Chen <chen.bo@intel.com>
A failure appeared in AArch64 musl cross build, after upgrading rust
to v1.47.0. A symbol "strrchr" was missing while linking against
static libfdt.a.
The issue could be caused by missing symbol(s) in new rust toolchain.
This fix pins the rust version in this cross build action to a stable-
enough version. Further upgrade will be done manually after testing.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
"Using a stable sort consumes more memory and cpu cycles. Because values
which compare equal are identical, preserving their relative order (the
guarantee that a stable sort provides) means nothing, while the extra
costs still apply."
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This is a new clippy check introduced in 1.47 which requires the use of
the matches!() macro for simple match blocks that return a boolean.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Instead of having the hypervisor crate embedding Cloud-Hypervisor forks
from the rust-vmm project, it's more appropriate to leave the rust-vmm
references in the hypervisor crate, and have the root Cargo.toml being
patched.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This changeset extends the documentation with the UEFI and Windows
related info. The focus is on providing consumer with a minimum
necessary and proper piece of the information to enter the features
quickly. While UEFI is a cross platform topic, it is a required
prerequisite for the Windows usage.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
The OneRegister literally means "one (arbitrary) register". Just call it
"Register" instead. There is no need to inherit KVM's naming scheme in
the hypervisor agnostic code.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
The goal here is to replace anywhere possible a virtio structure
with a "C, packed" representation by a "C" representation. Some
virtio structures are not expected to be packed, therefore there's
no reason for using the more restrictive "C, packed" representation.
This is important since "packed" representation can still cause
undefined behaviors with Rust 2018.
By removing the need for "packed" representation, we can simplify a
bit of code by deriving the Serialize trait without writing the
implementation ourselves.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Small patch creating a dedicated `block_io_uring_is_supported()`
function for the non-io_uring case, so that we can simplify the
code in the DeviceManager.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The purpose of that step is to make sure each commit builds. The `check`
command is much faster for that purpose.
On my 32-core machine `cargo check --all` takes around 25 seconds while
`cargo build --all` takes around 35 seconds, so that's quite a bit of
time saving there.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Since all unit and integration tests are run inside containers because
they are called from dev_cli.sh, they always run as root. That's why
both unit and integration scripts can be simplified as they don't need
to apply specific capabilities and run cargo tests in a dedicated 'kvm'
group.
Fixes#1683
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Windows will write zero to the IOAPIC version register as part of
probing if the device is there.
Fixes: #1791
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The existing virtio-blk hotplug test is extended by removing and
re-adding the virtio-blk device. This ensures the unplug/re-plug
feature is properly tested.
Fixes#1809
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The existing virtio-net hotplug test is extended by removing and
re-adding the virtio-net device. This ensures the unplug/re-plug
feature is properly tested.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Without the unlink(2) syscall being allowed, Cloud-Hypervisor crashes
when we remove a virtio-vsock device that has been previously added.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
It's important to ensure Cloud-Hypervisor's Git tree is bisectable, and
that's why this commit extends the existing build.yaml Github action.
This will validate that each commit from an incoming pull request can be
built.
Fixes#1808
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The Cloud-Hypervisor fork of the vhost crate contains one small
additional patch compared to the rust-vmm upstream version, meant for
increasing the connection timeout.
This patch is intended to be merged in order to check if it helps our CI
fixing the vhost-user-blk flakes that we've been observing recently.
If it fixes it, we'll submit a similar patch upstream and switch back to
the upstream vhost crate, otherwise we'll simply switch back to the
upstream crate, discarding this patch.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Because of the PCI refactoring that happened in the previous commit
d793cc4da3, the ability to fully remove a
PCI device was altered.
The refactoring was correct, but the usage of a generic function to pass
the same reference for both BusDevice, PciDevice and Any + Send + Sync
causes the Arc::ptr_eq() function to behave differently than expected,
as it does not match the references later in the code. That means we
were not able to remove the device reference from the MMIO and/or PIO
buses, which was leading to some bus range overlapping error once we
were trying to add a device again to the previous range that should have
been removed.
Fixes#1802
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Run loop in hypervisor needs a callback mechanism to access resources
like guest memory, mmio, pio etc.
VmmOps trait is introduced here, which is implemented by vmm module.
While handling vcpuexits in run loop, this trait allows hypervisor
module access to the above mentioned resources via callbacks.
Signed-off-by: Praveen Paladugu <prapal@microsoft.com>
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
A new version of vm-memory was released upstream which resulted in some
components pulling in that new version. Update the version number used
to point to the latest version but continue to use our patched version
due to the fix for #1258
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The PMEM support has an option called "discard_writes" which when true
will prevent changes to the device from hitting the backing file. This
is trying to be the equivalent of "readonly" support of the block
device.
Previously the memory of the device was marked as KVM_READONLY. This
resulted in a trap when the guest attempted to write to it resulting a
VM exit (and recently a warning). This has a very detrimental effect on
the performance so instead make "discard_writes" truly CoW by mapping
the memory as `PROT_READ | PROT_WRITE` and using `MAP_PRIVATE` to
establish the CoW mapping.
Fixes: #1795
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When all refblocks are consumed, the loop looking for the first free
cluster would access the element at refcounts[refcounts.len()], which is
out of bounds. Modify the free cluster search loop to check that the
index is in bounds before accessing it.
Tested-by: Daniel Verkamp <dverkamp@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Commit-Queue: Daniel Verkamp <dverkamp@chromium.org>
(cherry picked from crosvm commit f21572c7187c8beb9c6bfea6446351ae93200d01)
Fixes: #1792
Signed-off-by: Bo Chen <chen.bo@intel.com>
The virtio-balloon change the memory size is asynchronous.
VirtioBalloonConfig.actual of balloon device show current balloon size.
This commit add memory_actual_size to vm.info to show memory actual size.
Signed-off-by: Hui Zhu <teawater@antfin.com>
When the destination mode is physical, the destination field should
only be defined through bits 56-59, as defined in the IOAPIC spec. But
from the APIC specification, the APIC ID is always defined on 8 bits no
matter which destination mode is selected. That's why we always retrieve
the destination field based on bits 56-63.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This patch adds two required dependencies to fuzz/Cargo.toml, and fixes
the building error on the 'block' fuzzer.
Signed-off-by: Bo Chen <chen.bo@intel.com>
Now that Docker images are automatically generated for both amd64 and
arm64 architectures, there's no need to generate the arm64 image locally
on the ARM CI during a CI run. The image should be available from
DockerHub instead.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Write to the exit_evt EventFD which will trigger all the devices and
vCPUs to exit. This is slightly cleaner than just exiting the process as
any temporary files will be removed.
Fixes: #1242
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This commit moves back to the branch "virtio-fs-dev" from virtiofsd, as
we figured the changes needed to use this branch and the requirements
from the new meson build from QEMU.
It updates the container version to ensure the dev_cli.sh script will
rely on the latest container which contains the needed packages.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By fixing the Dockerfile, we have now finalized the automated generation
of the Docker images for both architectures (amd64 and arm64).
Fixes#953
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This patch adds the missing the `iommu` and `id` option for
`VmAddDevice` in the openApi yaml to respect the internal data structure
in the code base. Also, setting the `id` explicitly for VFIO device
hotplug is required for VFIO device unplug through openAPI calls.
Signed-off-by: Bo Chen <chen.bo@intel.com>
According to openAPI specification [1], the format for `integer` types
can be only `int32` or `int64`, unsigned and 8-bits integers are not
supported.
This patch replaces `uint64` with `int64`, `uint32` with `int32` and
`uint8` with `int32`.
[1]: https://swagger.io/specification/#data-types
Signed-off-by: Julio Montes <julio.montes@intel.com>
MsiInterruptGroup doesn't need to know the internal field names of
InterruptRoute. Introduce two helper functions to eliminate references
to irq_fd. This is done similarly to the enable and disable helper
functions.
Also drop the pub keyword from InterruptRoute fields. It is not needed
anymore.
No functional change.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
In order to support both amd64 and arm64, we rely on the TARGETARCH
variable that is passed from the docker buildx command, based on the
platform used to build the container image.
There is no way to rely directly on $(uname -m) to assign a variable
with the correct x86_64 or aarch64 values we're looking for. Both ENV
and ARG don't evaluate the command, which means they see it as a simple
string. Using RUN is the only way to evaluate a command.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to build virtiofsd from the latest build system, the Python
package python3-setuptools is required.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The idea is to rely on this new Github Action to detect when the
Dockerfile is updated after a push to the master branch on the
repository.
Once triggered, this action builds the Docker image for both
linux/amd64 and linux/arm64 platforms, and updates it directly
on Docker Hub.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
According to openAPI specification[1], the format for `integer` types
can be only `int32` or `int64`, unsigned integers are not supported.
This patch replaces `uint64` with `int64`.
[1]: https://swagger.io/specification/#data-types
Signed-off-by: Julio Montes <julio.montes@intel.com>
In order to speed up the Linux boot (so as to avoid it having to scan a
large number of pages) place the MP table directly after the SMBIOS
table if there is sufficient room. The start address of the SMBIOS table
is one of the three (and the largest) location that the MP table can
also be located at.
Before:
[ 0.000399] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT
[ 0.014945] check: Scanning 1 areas for low memory corruption
After:
[ 0.000284] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT
[ 0.000421] found SMP MP-table at [mem 0x000f0090-0x000f009f]
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
There is no point in manually dropping the lock for gsi_msi_routes then
instantly grabbing it again in set_gsi_routes.
Make set_gsi_routes take a reference to the routing hashmap instead.
No functional change intended.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
The MTRR feature was missing from the CPUID, which is causing the guest
to ignore the MTRR settings exposed through dedicated MSRs.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Removing the ISA DMA configurations prevents the kernel from accessing
the port I/O 0x87, which was generating the following warning:
WARN:vmm/src/cpu.rs:378 -- Guest PIO read to unregistered address 0x87
Removing the TELCLOCK configuration prevents the kernel from accessing
the port I/O reserved for the memory manager, which was causing the
following warning:
WARN:vmm/src/memory_manager.rs:289 -- Unexpected offset for accessing
memory manager device: 15
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The ::new() does very little beyond trying to open the /dev/kvm device
so provide a hint to the user about what has gone wrong.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
As discussed in #1707, the `vcpu` thread can be stalled when using
`--serial tty`. To workaround that issue, this patch enforces to resize
the pipe size to 256K when we capture the stdout/stderr of the
cloud-hypervisor child process in the integration tests. Note that the
pipe size (256K) is chosen based on the output size of our integration
tests at this point, which may need to be increased in the future.
Signed-off-by: Bo Chen <chen.bo@intel.com>
By looking at Linux kernel boot time, we identified that a lot of time
was spent registering and unregistering IRQ fds to KVM. This is not
efficient and certainly not a wrong behavior from the Linux kernel,
but rather a problem with the Cloud-Hypervisor's implementation of
MSI-X.
The way to fix this issue is by ensuring the initial conditions are
correct, which means the entire MSI-X vector table must be disabled
and masked. Additionally, each vector must be individually masked.
With these correct conditions, Linux won't start masking interrupt
vectors, and later unmask them since they will be seen as masked from
the beginning. This means the OS will simply have to unmask them when
needed, avoiding the extra operation.
Another aspect of this patch is to prevent Cloud-Hypervisor from
enabling (by registering IRQ fd) all vectors when either the global
'mask' or 'enable' bits are set. Instead, we can simply let the mask()
and unmask() operations take care of it if needed.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Since Cloud-Hypervisor currently support one single PCI bus, we must
reflect this through the MCFG table, as it advertises the first bus and
the last bus available. In this case both are bus 0.
This patch saves quite some time during guest kernel boot, as it
prevents from checking each bus for available devices.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
When created, the IOAPIC entries should be masked, as it is the guest's
responsibility (FW and/or OS) to unmask them if/when necessary.
This patch saves a full round of port I/O writes from the guest to the
IOAPIC, meant for masking the unmasked entries.
Because they're now masked, the entries are not enabled, which means
they are not connected from a KVM perspective, saving from unneeded
registration/unregistration of the irq fds.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
While we figure out the details on how to correctly build virtiofsd from
the latest rebase from the branch "virtio-fs-dev" (which now relies on
QEMU's new build system), let's fix the CI by relying on an older branch
which still relies on the previous build system.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The states of GIC should be part of the VM states. This commit
enables the AArch64 VM states save/restore by adding save/restore
of GIC states.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Currently for AArch64, the GICv3-ITS is tried to be created first
when PCI is not needed, which is unnecessary. This commit fixes
the problem.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
The definition of libc::SYS_ftruncate on AArch64 is different
from that on x86_64. This commit unifies the previously hard-coded
syscall number for AArch64.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
`KVM_GET_REG_LIST` ioctl is needed in save/restore AArch64 vCPU.
Therefore we whitelist this ioctl in seccomp.
Also this commit unifies the `SYS_FTRUNCATE` syscall for x86_64
and AArch64.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Similarly as the VM booting process, on AArch64 systems,
the vCPUs should be created before the creation of GIC. This
commit refactors the vCPU save/restore code to achieve the
above-mentioned restoring order.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Since calling `KVM_GET_ONE_REG` before `KVM_VCPU_INIT` will
result in an error: Exec format error (os error 8). This commit
decouples the vCPU init process from `configure_vcpus`. Therefore
in the process of restoring the vCPUs, these vCPUs can be
initialized separately before started.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
The construction of `GICR_TYPER` register will need vCPU states.
Therefore this commit adds methods to extract saved vCPU states
from the cpu manager.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Unlike x86_64, the "interrupt_controller" in the device manager
for AArch64 is only a `Gic` object that implements the
`InterruptController` to provide the interrupt delivery service.
This is not the real GIC device so that we do not need to save
its states. Also, we do not need to insert it to the device_tree.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
The value of GIC register `GICR_TYPER` is needed in restoring
the GIC states. This commit adds a field in the GIC device struct
and a method to construct its value.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
In AArch64 systems, the state of GIC device can only be
retrieved from `KVM_GET_DEVICE_ATTR` ioctl. Therefore to implement
saving/restoring the GIC states, we need to make sure that the
GIC object (either the file descriptor or the device itself) can
be extracted after the VM is started.
This commit refactors the code of GIC creation by adding a new
field `gic_device_entity` in device manager and methods to set/get
this field. The GIC object can be therefore saved in the device
manager after calling `arch::configure_system`.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This commit adds a function which allows to save RDIST pending
tables to the guest RAM, as well as unit test case for it.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This commit adds the unit test cases for getting/setting the GIC
distributor, redistributor and ICC registers.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This commit implements the `get_device_attr` method for the
`KVM_GET_DEVICE_ATTR` ioctl. This ioctl will be used in retrieving
the GIC status.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This commit moves the GIC-related code to a separate module.
Therefore the implementation of GIC registers can be introduced
to the new module.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Adds 3 more unit test cases for AArch64:
*save_restore_core_regs
*save_restore_system_regs
*get_set_mpstate
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This commit adds methods to save/restore AArch64 vCPU registers,
including:
1. The AArch64 `VcpuKvmState` structure.
2. Some `Vcpu` trait methods of the `KvmVcpu` structure to
enable the save/restore of the AArch64 vCPU states.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This commit ports code from firecracker and refactors the existing
AArch64 code as the preparation for implementing save/restore
AArch64 vCPU, including:
1. Modification of `arm64_core_reg` macro to retrive the index of
arm64 core register and implemention of a helper to determine if
a register is a system register.
2. Move some macros and helpers in `arch` crate to the `hypervisor`
crate.
3. Added related unit tests for above functions and macros.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
virtio-mem device would use 'VIRTIO_MEM_F_ACPI_PXM' to add memory to NUMA
node, which MUST be existed, otherwise it will be assigned to node id 0,
even if user specify different node id.
According ACPI spec about Memory Affinity Structure, system hardware
supports hot-add memory region using 'Hot Pluggable | Enabled' flags.
Signed-off-by: Jiangbo Wu <jiangbo.wu@intel.com>
Use zone.host_numa_node to create memory zone, so that memory zone
can apply memory policy in according with host numa node ID
Signed-off-by: Jiangbo Wu <jiangbo.wu@intel.com>
If after the creation of the self-spawned backend, the VMM cannot create
the corresponding vhost-user frontend, the VMM must kill the freshly
spawned process in order to ensure the error propagation can happen.
In case the child process would still be around, the VMM cannot return
the error as it waits onto the child to terminate.
This should help us identify when self-spawned failures are caused by a
connection being refused between the VMM and the backend.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
When the VMM is terminated by receiving a SIGTERM signal, the signal
handler thread must be able to invoke ioctl(TCGETS) and ioctl(TCSETS)
without error.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Build testing of changes happens on GitHub actions and the integration
tests will build the binary (with different feature flags) again. So
these earlier build operations are just wasted time on the critical
path.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Only use `cloud-hypervisor` when referring to the binary itself and
prefer Cloud Hypervisor when referring to the project.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Some operations complete directly after they have been submitted, which
means they are not submitted asynchronously and therefore they don't
generate any ioevent. This is the reason why we are not processing some
of the completed operations, which leads to some unpredictable
behaviors.
Forcing all io_uring operations submitted to the SQE to be asynchronous
helps simplifying the code as it ensures the completion of every
operation will generate an ioevent, therefore no operation is missed.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that virtio-mem supports reboot, we extend the existing integration
tests to validate the amount of RAM after reboot is the same as before
the reboot, but also that we can still resize down the VM or the memory
zone after the reboot.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The missing syscall rt_sigprocmask(2) was triggered for the musl build
upon rebooting the VM, and was causing the VM to be killed.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to facilitate the memory hotplug for our users, let's enable
the automatic memory onlining in our guest kernel by activating the
kernel config option 'CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE'.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on all the preparatory work achieved through previous commits,
this patch updates the 'hotplugged_size' field for both MemoryConfig and
MemoryZoneConfig structures when either the whole memory is resized, or
simply when a memory zone is resized.
This fixes the lack of support for rebooting a VM with the right amount
of memory plugged in.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Adding a new field to VirtioMemZone structure, as it lets us associate
with a particular virtio-mem region the amount of memory that should be
plugged in at boot.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This patch simplifies the code as we have one single Option for the
VirtioMemZone. This also prepares for storing additional information
related to the virtio-mem region.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Add the new option 'hotplugged_size' to both --memory-zone and --memory
parameters so that we can let the user specify a certain amount of
memory being plugged at boot.
This is also part of making sure we can store the virtio-mem size over a
reboot of the VM.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit gives the possibility to create a virtio-mem device with
some memory already plugged into it. This is preliminary work to be
able to reboot a VM with the virtio-mem region being already resized.
Signed-off-by: Hui Zhu <teawater@antfin.com>
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that e820 tables are created from the 'boot_guest_memory', we can
simplify the memory manager code by adding the virtio-mem regions when
they are created. There's no need to wait for the first hotplug to
insert these regions.
This also anticipates the need for starting a VM with some memory
already plugged into the virtio-mem region.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to differentiate the 'boot' memory regions from the virtio-mem
regions, we store what we call 'boot_guest_memory'. This is useful to
provide the adequate list of regions to the configure_system() function
as it expects only the list of regions that should be exposed through
the e820 table.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Extend the existing test to validate that each NUMA node gets assigned
the right amount of memory after each memory zone has been resized.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that we can resize each memory zone independently, this commit
extends the memory zone related test by validating 'vm.resize-zone'
works correctly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The virtio-mem driver is generating some warnings regarding both size
and alignment of the virtio-mem region if not based on 128MiB:
The alignment of the physical start address can make some memory
unusable.
The alignment of the physical end address can make some memory
unusable.
For these reasons, the current patch enforces virtio-mem regions to be
128MiB aligned and checks the size provided by the user is a multiple of
128MiB.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that virtio-mem device accept a guest NUMA node as parameter, we
retrieve this information from the list of NUMA nodes. Based on the
memory zone associated with the virtio-mem device, we obtain the NUMA
node identifier, which we provide to the virtio-mem device.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Implement support for associating a virtio-mem device with a specific
guest NUMA node, based on the ACPI proximity domain identifier.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
For more consistency and help reading the code better, this commit
renames all 'virtiomem*' variables into 'virtio_mem*'.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By testing manually the memory resizing through virtio-mem, several
missing syscalls have been identified.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Implement a new VM action called 'resize-zone' allowing the user to
resize one specific memory zone at a time. This relies on all the
preliminary work from the previous commits to resize each virtio-mem
device independently from each others.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By adding a new parameter 'id' to the virtiomem_resize() function, we
prepare this function to be usable for both global memory resizing and
memory zone resizing.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
It's important to return the region covered by virtio-mem the first time
it is inserted as the device manager must update all devices with this
information.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the previous code changes, we can now update the MemoryManager
code to create one virtio-mem region and resizing handler per memory
zone. This will naturally create one virtio-mem device per memory zone
from the DeviceManager's code which has been previously updated as well.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In anticipation for resizing support of an individual memory zone,
this commit introduces a new option 'hotplug_size' to '--memory-zone'
parameter. This defines the amount of memory that can be added through
each specific memory zone.
Because memory zone resize is tied to virtio-mem, make sure the user
selects 'virtio-mem' hotplug method, otherwise return an error.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Both MemoryManager and DeviceManager are updated through this commit to
handle the creation of multiple virtio-mem devices if needed. For now,
only the framework is in place, but the behavior remains the same, which
means only the memory zone created from '--memory' generates a
virtio-mem region that can be used for resize.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to anticipate the need for storing memory regions along with
virtio-mem information for each memory zone, we create a new structure
MemoryZone that will replace Vec<Arc<GuestRegionMmap>> in the hash map
MemoryZones.
This makes thing more logical as MemoryZones becomes a list of
MemoryZone sorted by their identifier.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Inject CPUID leaves for advertising KVM HyperV support when the
"kvm_hyperv" toggle is enabled. Currently we only enable a selection of
features required to boot.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Currently we don't need to do anything to service these exits but when
the synthetic interrupt controller is active an exit will be triggered
to notify the VMM of details of the synthetic interrupt page.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The Windows virtio block driver puts multiple data descriptors between
the header and the status footer. To handle this when parsing iterate
over the descriptor chain until the end is reached accumulating the
address and length pairs in a vector. For execution iterate over the
vector and make sequential reads from the disk for each data descriptor.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In anticipation for supporting multiple virtio descriptors, let's make
sure the read/write operations are performed with vectored I/O.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Some of the io_uring setup happens upon activation of the virtio-blk
device, which is initially triggered through an MMIO VM exit. That's why
the vCPU threads must authorize io_uring related syscalls.
This commit ensures the virtio-blk io_uring implementation can be used
along with the seccomp filters enabled.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We observed CI instability for the past couple of days. This
instability is confirmed to be a result of incomplete seccomp
filters. Given the filter on 'virtio_vsock' is recently added and
is missing 'brk', it is likely to be the root cause of the
instability.
Signed-off-by: Bo Chen <chen.bo@intel.com>
Extract common code for adding devices to the PCI bus into its own
function from the VFIO and VIRTIO code paths.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This removes the dependency of the pci crate on the devices crate which
now only contains the device implementations themselves.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Extending the Cloud-Hypervisor CI to allow for testing SGX on a
dedicated machine where special image and kernels are ready.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The new function already checks if the API version is compatible. There
is no need to expose the get_api_version function to code outside
hypervisor crate.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
In our build-script (build.rs), we won't set the environment variable
'BUILD_VERSION' when the 'git describe' command failed (e.g. when the
current source tree does not contain git information). This patch added
a fall back path where the default value of 'BUILD_VERSION' is based on
the 'cloud-hypervisor' crate version.
Fixes: #1669
Signed-off-by: Bo Chen <chen.bo@intel.com>
The goal of this commit is to rename the existing NUMA option 'id' with
'guest_numa_id'. This is done without any modification to the way this
option behaves.
The reason for the rename is caused by the observation that all other
parameters with an option called 'id' expect a string to be provided.
Because in this particular case we expect a u32 representing a proximity
domain from the ACPI specification, it's better to name it with a more
explicit name.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the recent changes regarding how to describe NUMA nodes to the
guest, the documentation related to memory zones and NUMA has been
updated accordingly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Since both --memory-zone and --numa parameters have been updated with
addition and removal of multiple options, this commit updates the
related integration tests to ensure they are still valid.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The way to describe guest NUMA nodes has been updated through previous
commits, letting the user describe the full NUMA topology through the
--numa parameter (or NumaConfig).
That's why we can remove the deprecated and unused 'guest_numa_node'
option.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the previous changes introducing new options for both memory
zones and NUMA configuration, this patch changes the behavior of the
NUMA node definition. Instead of relying on the memory zones to define
the guest NUMA nodes, everything goes through the --numa parameter. This
allows for defining NUMA nodes without associating any particular memory
range to it. And in case one wants to associate one or multiple memory
ranges to it, the expectation is to describe a list of memory zone
through the --numa parameter.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This new option provides a new way to describe the memory associated
with a NUMA node. This is the first step before we can remove the
'guest_numa_node' option from the --memory-zone parameter.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that we have an identifier per memory zone, and in order to keep
track of the memory regions associated with the memory zones, we create
and store a map referencing list of memory regions per memory zone ID.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In anticipation for allowing memory zones to be removed, but also in
anticipation for refactoring NUMA parameter, we introduce a mandatory
'id' option to the --memory-zone parameter.
This forces the user to provide a unique identifier for each memory zone
so that we can refer to these.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Split the block device implementation into code that be used in common
between multiple different virtio device implementations.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In order to simplify the transition to VirtioCommon and to avoid needing
to set empty fields derive Default for VirtioCommon.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Rearrange the code to match other devices which makes it easier to prep
for sharing this between other devices.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Move the if-let for the taps later which makes the earlier activation
code identical to other devices.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Introduce VirtioCommon to help remove duplicated functionality and state
between implementations of VirtioDevice. Initially it is only handling
feature acknowledgement and testing.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This patch ported many tests to the new methodology, where the guest log
will be printed only when the test is failing.
Things to finish in follow-up PRs:
1. Special tests not ported yet include: test_reboot,
test_bzimage_reboot, test_serial_null(), test_serial_tty(),
test_serial_file(), test_virtio_console(), test_console_file(),
and test_simple_launch.
2. Few direct calls to 'Command::new(clh_command("cloud-hypervisor"))',
which is still printing the guest console
Signed-off-by: Bo Chen <chen.bo@intel.com>
Writing some new documentation to help users understand how the guest
memory can be described through Cloud-Hypervisor parameters.
Fixes#1659
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By extending the existing NUMA integration test, this commit validates
the proper distances between NUMA nodes are exposed to the guest.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By introducing the SLIT (System Locality Distance Information Table), we
provide the guest with the distance between each node. This lets the
user describe the NUMA topology with a lot of details so that slower
memory backing the VM can be exposed as being further away from other
nodes.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the NumaConfig which now provides distance information, we can
internally update the list of NUMA nodes with the exact distances they
should be located from other nodes.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By introducing 'distances' option, we let the user describe a list of
destination NUMA nodes with their associated distances compared to the
current node (defined through 'id').
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Extend the existing NUMA integration to validate that specifying CPUs
for each NUMA node gets propagated to the guest.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the list of CPUs related to each NUMA node, Processor Local
x2APIC Affinity structures are created and included into the SRAT table.
This describes which CPUs are part of each node.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Relying on the list of CPUs defined through the NumaConfig, this patch
will update the internal list of CPUs attached to each NUMA node.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Through this new parameter, we give users the opportunity to specify a
set of CPUs attached to a NUMA node that has been previously created
from the --memory-zone parameter.
This parameter will be extended in the future to describe the distance
between multiple nodes.
For instance, if a user wants to attach CPUs 0, 1, 2 and 6 to a NUMA
node, here are two different ways of doing so:
Either
./cloud-hypervisor ... --numa id=0,cpus=0-2:6
Or
./cloud-hypervisor ... --numa id=0,cpus=0:1:2:6
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This new test validates the guest OS can find the NUMA nodes which have
been defined by the user through the CLI, and that the right amount of
memory is associated with each node.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The SRAT table (System Resource Affinity Table) is needed to describe
NUMA nodes and how memory ranges and CPUs are attached to them.
For now it simply attaches a list of Memory Affinity structures based on
the list of NUMA nodes created from the VMM.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the 'guest_numa_node' option, we create and store a list of
NUMA nodes in the MemoryManager. The point being to associate a list of
memory regions to each node, so that we can later create the ACPI tables
with the proper memory range information.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
With the introduction of this new option, the user will be able to
describe if a particular memory zone should belong to a specific NUMA
node from a guest perspective.
For instance, using '--memory-zone size=1G,guest_numa_node=2' would let
the user describe that a memory zone of 1G in the guest should be
exposed as being associated with the NUMA node 2.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Given that ACPI uses u32 as the type for the Proximity Domain, we can
use u32 instead of u64 as the type for 'host_numa_node' option.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
"struct MemoryConfig" has balloon_size but not in MemoryConfig
of cloud-hypervisor.yaml.
This commit adds it.
Signed-off-by: Hui Zhu <teawater@antfin.com>
Let's narrow down the limitation related to mbind() by allowing shared
mappings backed by a file backed by RAM. This leaves the restriction on
only for mappings backed by a regular file.
With this patch, host NUMA node can be specified even if using
vhost-user devices.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Relying on the new option 'host_numa_node' from the 'memory-zone'
parameter, the user can now define which NUMA node from the host
should be used to back the current memory zone.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Since memory zones have been introduced, it is now possible for a user
to specify multiple backends for the guest RAM. By adding a new option
'host_numa_node' to the 'memory-zone' parameter, we allow the guest RAM
to be backed by memory that might come from a specific NUMA node on the
host.
The option expects a node identifier, specifying which NUMA node should
be used to allocate the memory associated with a specific memory zone.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The flag 'mergeable' should only apply to the entire guest RAM, which is
why it is removed from the MemoryZoneConfig as it is defined as a global
parameter at the MemoryConfig level.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The 'cmdline' parameter should not be required as it is not needed when
the 'kernel' parameter is the rust-hypervisor-fw, which means the kernel
and the associated command line will be found from the EFI partition.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Factorize the codepath between simple memory and multiple memory zones.
This simplifies the way regions are memory mapped, as everything relies
on the same codepath. This is performed by creating a memory zone on the
fly for the specific use case where --memory is used with size being
different from 0. Internally, the code can rely on memory zones to
create the memory regions forming the guest memory.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
After the introduction of user defined memory zones, we can now remove
the deprecated 'file' option from --memory parameter. This makes this
parameter simpler, letting more advanced users define their own custom
memory zones through the dedicated parameter.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
User defined memory regions can now support being snapshot and restored,
therefore this commit removes the restrictions that were applied through
earlier commit.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By factorizing a lot of code into create_ram_region(), this commit
achieves the simplification of the restore codepath. Additionally, it
makes user defined memory zones compatible with snapshot/restore.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
First thing, this patch introduces a new function to identify if a file
descriptor is linked to any hard link on the system. This can let the
VMM know if the file can be accessed by the user, or if the file will
be destroyed as soon as the VMM releases the file descriptor.
Based on this information, and associated with the knowledge about the
region being MAP_SHARED or not, the VMM can now decide to skip the copy
of the memory region content. If the user has access to the file from
the filesystem, and if the file has been mapped as MAP_SHARED, we can
consider the guest memory region content to be present in this file at
any point in time. That's why in this specific case, there's no need for
performing the copy of the memory region content into a dedicated file.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Let's not assume that a backing file is going to be the result from
a snapshot for each memory region. These regions might be backed by
a file on the host filesystem (not a temporary file in host RAM), which
means they don't need to be copied and stored into dedicated files.
That's why this commit prepares for further changes by introducing an
optional PathBuf associated with the snapshot of each memory region.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
There will be some cases where the implementation of the snapshot()
function from the Snapshottable trait will require to modify some
internal data, therefore we make this possible by updating the trait
definition with snapshot(&mut self).
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Adding a small test to validate that user defined memory zones work as
expected when using --memory-zone option.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In case the memory size is 0, this means the user defined memory
zones are used as a way to specify how to back the guest memory.
This is the first step in supporting complex use cases where the user
can define exactly which type of memory from the host should back the
memory from the guest.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In anticipation for the need to map part of a file with the function
create_ram_region(), it is extended to accept a file offset as argument.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In case the provided backing file is an actual file and not a directory,
we should not truncate it, as we expect the file to already be the right
size.
This change will be important once we try to map the same file through
multiple memory mappings. We can't let the file be truncated as the
second mapping wouldn't work properly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Introducing a new CLI option --memory-zone letting the user specify
custom memory zones. When this option is present, the --memory size
must be explicitly set to 0.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
It is otherwise seems to be able to cause resource conflicts with
Windows APCI_HAL. The OS might do a better job on assigning resources
to this device, withouth them to be requested explicitly. 0xcf8 and
0xcfc are only what is certainly needed for the PCI device enumeration.
Signed-off-by: Anatol Belski <anatol.belski@microsoft.com>
Use of backing file is deprecated hence use the `hugepages` field.
Also use the `boot` field for specifying number of cpus
Signed-off-by: Amey Narkhede <ameynarkhede02@gmail.com>
We may need to store hypervisor speciific data to the VM. This support is
needed for Microsoft hyperv implementations. This patch introduces two
new definitions to Vm trait and implements for KVM.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Problem with the previous solution was that Cleanup stage was not
executed when previous stages failed. We fix this by adding a post
section that executes always after the Aarch64 build completed, no
matter if it has failed, succeeded or been aborted.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
"debug!" marco is used in virtio-devices/src/epoll_helper.rs. When"-vvv"
and "--log-file" option was specified, the missing "SYS_write" rule
caused a "bad system call" crash.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Cleanup of the Aarch64 machine can't be done as part of the parallel
stage as this is often skipped. When the build is aborted because
another parallel stage failed, the post actions are simply not
performed. That's why we need a dedicated stage, out of the parallel
ones, to cleanup the Aarch64 machine.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Use updated --memory option parameters. The field `size` needs
M/G suffix. Without the suffix cloud-hypervisor panics at
src/main.rs:353
Also the use of backing file is deprecated so use `shared` field
Signed-off-by: Amey Narkhede <ameynarkhede02@gmail.com>
As we never join the spawned virtio-devices worker threads, the error
returned from each worker thread is lost. For now, we simply print out
the error from each worker thread.
Fixes: #1551
Signed-off-by: Bo Chen <chen.bo@intel.com>
From the experiments of running integration tests on my local machine,
auditd occationally reported the 'brk' syscall is needed for the
'virtio-rng' worker thread.
Signed-off-by: Bo Chen <chen.bo@intel.com>
This patch adds the seccomp filter list for the virtio_net thread, while
the list was already added for the virtio_net_ctl thread.
Partially fixes: #925
Signed-off-by: Bo Chen <chen.bo@intel.com>
The current seccomp filter for virtio-net is actually for the worker
thread 'virtio_net_ctl' (not the actual worker thread
'virtio_net'). This patch introduces changes to distinguish those two
worker threads and seccomp filters.
Signed-off-by: Bo Chen <chen.bo@intel.com>
We recently added the ability for the gnu and musl workers to retry if
integration test were not passing, relying on some simple Jenkins
options. Unfortunately, this is not working as expected as the retries
never pass either. The suspected reason is the machine itself, which
might be scheduled on some specific hardware, which makes our VMs more
error prone.
Bottom line, on a faulty machine, the tests will always fail, therefore
there is no added value in retrying on the same machine.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Some OS might check for duplicates and bail out, if it can't create a
distinct mapping. According to ACPI 5.0 section 6.1.12, while _UID is
optional, it becomes required when there are multiple devices with the
same _HID.
Signed-off-by: Anatol Belski <ab@php.net>
Using the Rust Barrier mechanism, this patch forces each virtio device
to acknowledge they've been correctly paused before going further.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
If the build was aborted then the dev_cli.sh code that is responsible
for changing the file ownership will not get run. This results in the
failure to delete some of the files in the workspace.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The seccomp filters specific to the virtio-net threads must contain
dup() syscall now that we ported the epoll code to the EpollHelper.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Migrate virtio-net and vhost-user-net control queue to EpollHelper so
as to remove code duplication.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This required a bit of rearranging as it is not possible to call
prepare_daemon() inside a catch_unwind{} block.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Migrate all vhost-user devices to EpollHelper so as to remove code that
is duplicated between multiple virtio devices.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now the testing aspects are run inside a panic handler block rather than
inside a credibility TestBlock. If the test fails then the output from
the cloud-hypervisor binary is then presented.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This reduces the complexity of the test slightly. The PCI BDFs in the L1
needed changing as the block devices come before the network ones.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Migrate to EpollHelper so as to remove code that is duplicated between
multiple virtio devices.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Instead of passing only the event type through the handle_event()
callback, we make the trait slightly more generic by providing the
epoll event to each virtio device implementation.
This is particularly useful for vsock as it will need the event set.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Migrate to EpollHelper so as to remove code that is duplicated between
multiple virtio devices.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The seccomp filters specific to the virtio-console thread must contain
dup syscall now that we ported the epoll code to the EpollHelper.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Migrate to EpollHelper so as to remove code that is duplicated between
multiple virtio devices.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The brk syscall is not always called as the system might not need it.
But when it's needed from the API thread, this causes the thread to
terminate as it is not part of the authorized list of syscalls.
This should fix some sporadic failures on the CI with the musl build.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Even if the vhost-user-net device did implement all currently-defined
features, it would be very short-sighted to use ::all(), because if a
new feature was defined later, the device would start claiming to
implement it even though it didn't.
More practically, claiming to implement all features breaks using QEMU
with the cloud-hypervisor vhost-user-net backend, because QEMU will
negotiate VHOST_USER_PROTOCOL_F_SLAVE_REQ, and then break when the
communication channel isn't actually set up.
I wasn't sure exactly which features the backend should claim to
implement, though. Definitely MQ, and I'm fairly certain none of the
features I've ommitted are implemented. But I'm not sure about
REPLY_ACK. As far as I can tell it should be implemented entirely by
the vhost crate, with no cooperation required from the vhost-user-net
backend itself, so there should be no reason to let a frontend use it
if it wants to. But despite this, neither vhost-user-fs nor
vhost-user-blk claims to implement it.
Signed-off-by: Alyssa Ross <hi@alyssa.is>
Rather than using a credible TestBlock to capture the test assertions
instead use a catch_unwind block to catch the panic and turn
it into a Result<>.
If block panicked or the child binary had non-zero exit then, and only
then, print out the child output.
This results in a clearer test output with no interleaving.
Currently only test_counters is ported to this methodology to
demonstrate its use.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
`cargo test` runs every test in its own process and by default will
collect the output from the test and only print it if the test fails.
`--nocapture` instead prints any output from the test immediately and
does not collate it - this leads to interleaving.
`--nocapture` might be useful for local test debugging but is not
something that is helpful, and is instead unhelpful for our CI as a
whole.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Now that io-uring crate has been stabilized, let's move to the latest
release 0.4.0 from crates.io.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Migrate to EpollHelper so as to remove code that is duplicated between
multiple virtio devices.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Packaging the cloud-hyperviosr binary is useful for enterprise use.
This patch adds the spec file for x86_64. Aarch64 support is not added
yet.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Both gnu and musl workers will retry integration tests up to 3 times if
they fail. This should give us a better pass rate, without having to
restart the entire build every time a single glitch happens.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Add mprotect to API thread rules. Prevent the VMM is
killed when it is used.
Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
This fixes `cargo vendor` throwing an error
```
$ cargo vendor
error: failed to sync
Caused by:
found duplicate version of package `vfio-bindings v0.2.0` vendored from two sources:
source 1: https://github.com/rust-vmm/vfio-bindings#f08cbcbf
source 2: registry `https://github.com/rust-lang/crates.io-index`
```
Both sources are indeed same, the conflict is only cause by the
different URLs.
Signed-off-by: Anatol Belski <ab@php.net>
Based on the way KVM_GET_MSRS and KVM_SET_MSRS work, both function are
very unlikely to fail, as they simply stop looping through the list of
MSRs as soon as getting or setting one fails. This is causing some
issues with the snapshot/restore feature, as on some platforms, we only
save a subset of the list of MSRs, leading to unproper way of saving the
VM.
The way to address this issue is by checking the number of MSRs get/set
matches the expected amount from the list. In case it does not match, we
simply ignore the failing MSR and continue getting/setting the rest of
the list. By doing this by iterations, we end up getting/setting as many
MSRs as the platform can support.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Turns out that "when" blocks are permitted but not effectual in "post"
sections so instead use a script behaviour to make the message
conditional on build change.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Migrate to EpollHelper so as to remove code that is duplicated between
multiple virtio devices.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Currently any messages generated during the worker thread are not
shown anywhere as the thread is never join()ed on. Instead output the
error immediately.
For now only cover the subset where the work to port to EpollHandler
clashed with the seccomp filtering for virtio devices.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
We no longer download focal-server-cloudimg-arm64.img, checking it
lead to a failure if "work_loads" folder is new.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Migrate to EpollHelper so as to remove code that is duplicated between
multiple virtio devices.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Migrate to EpollHelper so as to remove code that is duplicated between
multiple virtio devices.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Increase the number of open files limit for the sandboxed process to the
maximum allowed in the system. The maximum is obtained by reading the
/proc/sys/fs/nr_open sysctl file, and the setting is done using the setrlimit
syscall. Failure to read or parse the nr_open file, or to set the rlimit
results in a panic.
Signed-off-by: Ricardo Koller <ricarkol@gmail.com>
This commit enables the test case for testing the basic function
of virtio_vsock (i.e. without the hotplug).
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This patch added the seccomp_filter module to the virtio-devices crate
by taking reference code from the vmm crate. This patch also adds
allowed-list for the virtio-block worker thread.
Partially fixes: #925
Signed-off-by: Bo Chen <chen.bo@intel.com>
This patch propagates the SeccompAction value from main to the
Vm struct constructor (i.e. Vm::new_from_memory_manager), so that we can
use it to construct the DeviceManager and CpuManager struct for
controlling the behavior of the seccomp filters for vcpu/virtio-device
worker threads.
Signed-off-by: Bo Chen <chen.bo@intel.com>
This patch extends the CLI option '--seccomp' to accept the 'log'
parameter in addition 'true/false'. It also refactors the
vmm::seccomp_filters module to support both "SeccompAction::Trap" and
"SeccompAction::Log".
Fixes: #1180
Signed-off-by: Bo Chen <chen.bo@intel.com>
This patch replaces the usage of 'SeccompLevel' with 'SeccompAction',
which is the first step to support the 'log' action over system
calls that are not on the allowed list of seccomp filters.
Signed-off-by: Bo Chen <chen.bo@intel.com>
This commit enables the `api_create_boot` case in the integration
test as the test for the Cloud Hypervisor API server functionality.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
By adding a new io_uring feature gate, we let the user the possibility
to choose if he wants to enable the io_uring improvements or not.
Since the io_uring feature depends on the availability on recent host
kernels, it's better if we leave it off for now.
As soon as our CI will have support for a kernel 5.6 with all the
features needed from io_uring, we'll enable this feature gate
permanently.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Instead of just logging error messages but continue the processing of
the queues, this patch returns errors right away. This allows for a
quicker detection of an error happening on the virtqueue.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In case the host supports io_uring and the specific io_uring options
needed, the VMM will choose the asynchronous version of virtio-blk.
This will enable better I/O performances compared to the default
synchronous version.
This is also important to note the VMM won't be able to use the
asynchronous version if the backend image is in QCOW format.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This introduces a new version of virtio-blk device. The default
virtio-blk provides synchronous processing of the queues, while this
new version relies on io_uring from the host kernel to provide an
asynchronous processing of the queues.
This new asynchronous version provides a huge performance improvement
compared to the default synchronous version.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Creates a dedicated function relying on io_uring crate to execute
io_uring specific requests.
Also creates a function for checking io_uring support on the host.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
PR#1511 introduced a `flock` command in order to let AArch64 CI
can be executed with multiple executors. However the command
```
(
echo "try to lock $WORKLOADS_DIR folder and update"
flock -x 12 && update_workloads
)
```
will introduce an abnormal behavior: If any error happened in
function `updated_workloads`, the sub-shell opened by the pair of
parentheses will be killed instead of the main shell, which is not
right.
This commit fixes this abnormal execution behaviour.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Cloud Hypervisor allows either the serial or virtio console to output to
TTY, but TTY input is pushed to both.
This is not correct. When Linux guest is configured to spawn TTYs on
both ttyS0 and hvc0, the user effectively issues the same commands twice
in different TTYs.
Fix this by only direct input to the one choice that is using host side
TTY.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Lock "work_loads" folder when one job is syncing files. If another job
arrives, wait until the lock is released.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
The command which is mentioned in the README is wrong. We have two
cofigs one for x86_64 and another for aarh64. Previously it was a single
config. After adding the configs the read me was not modified. This
patch fixes the issue.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
In order to follow the virtio-fs development, we rely on the latest
development branch regarding the virtio-fs daemon. It will be now build
from virtio-fs-dev branch from the virtio-fs maintainers repository.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Move to latest vhost version, as it contains the updated list of
vhost-user protocol features along with the updated list of virtio-fs
slave commands.
This will allow Cloud-Hypervisor to work with latest virtiofsd binary.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit fixes an "Bad syscall" error when shutting down the VM
on AArch64 by adding the SYS_unlinkat syscall to the seccomp
whitelist.
Signed-off-by: Jianyong Wu <jianyong.wu@arm.com>
EpollHelper allows the removal of much duplicated loop handling code and
instead the device specific even handling is delegated via an
implementation of EpollHelperHandler.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This is required for implementing virtio-net as the epoll RawFd must be
assigned into the NetQueuePair.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Ensure that the width of the I/O port is correctly set to 32-bits in the
generic address used for the X_PM_TMR_BLK. Do this by type
parameterising GenericAddress::io_port_address() fuction.
TEST=Boot with clocksource=acpi_pm and observe no errors in the dmesg.
Fixes: #1496
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Port virtio-block device to the new EpollHelper. This required moving
the queue EventFd ownership to BlockEpollHandler.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This is a helper for implementing the worker thread for virtio devices
and in particular handles special behaviour for pause and kill events.
The device specific event handling (for the queues themselves) is
delegated to a method invoked on a new EpollHelperHandler trait. This
method is passed the event as well as the EpollHelper so that it can
operate on the handler in order to manage events itself (required for
virtio-net.)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This is a counter exposed via an I/O port that runs at 3.579545MHz. Here
we use a hardcoded I/O and expose the details through the FADT table.
TEST=Boot Linux kernel and see the following in dmesg:
[ 0.506198] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add some documentation regarding the snapshot/restore feature which was
introduced earlier, explaining how the feature works and how to use it
properly.
Fixes#1490
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This bit should be set for the last 224us of each second. To be able to
fill this bit we need to use clock_gettime() to get the nanoseconds.
Also set the frequency bit to indicate we update at 32kHz.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
We store the device passthrough handler, so we should use it through our
internal API and only carry the passed through device configuration.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Shrink GICDevice trait to contain hypervisor agnostic API's only, which
are used in generating FDT.
Move all KVM specific logic into KvmGICDevice trait.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
This commit enables some mmio-related integration test cases on
AArch64, including:
* some vhost_user test cases
* virtio-blk test cases
* pmem test cases
Also this commit contains a bug fix in creating virtio-blk device.
Previously, when creating the FDT, the virtio-blk device was
labeled in the reverse order of address allocation.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Make set_gsi_routing take a list of IrqRoutingEntry. The construction of
hypervisor specific structure is left to set_gsi_routing.
Now set_gsi_routes, which is part of the interrupt module, is only
responsible for constructing a list of routing entries.
This further splits hypervisor specific code from hypervisor agnostic
code.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
That function is going to return a handle for passthrough related
operations.
Move create_kvm_device code there.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Add the basic infrastructure for fuzzing along with a qcow fuzzer ported
from crosvm and adapted to our code.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The only driver writable field in the virtio-block specification is the
writeback one. Check that the offset being written to is for that field
and update it.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The only driver writable field in the virtio-block specification is the
writeback one. Check that the offset being written to is for that field
and update it.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add a helper function to share code between implementations that can use
a slice accessible data structure for configuration data.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Remove the write_config() implementations that only generate a warning
as that is now done at the VirtioDevice level.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Not every virtio device has any config fields that can be read and most
have none that can be written to.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The retry order to create virtual GIC is GICv3-ITS, GICv3 and GICv2.
But there was not log message to show what was finally created.
The log messages also mute the warning for unused "log" crate.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
With recent changes (#1030#1084), the '/vm.remove-device' API is no
longer restricted to remove VFIO PCI devices. It now supports (almost)
all hotpluggable devices, e.g. disk/net/pmem/fs/vsock.
Signed-off-by: Bo Chen <chen.bo@intel.com>
It returns an hypervisor object depending on which hypervisor is
configured. Currently it only supports KVM.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
SGX expects the EPC region to be reported as "reserved" from the e820
table. This patch adds a new entry to the table if SGX is enabled.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The support for SGX is exposed to the guest through CPUID 0x12. KVM
passes static subleaves 0 and 1 from the host to the guest, without
needing any modification from the VMM itself.
But SGX also relies on dynamic subleaves 2 through N, used for
describing each EPC section. This is not handled by KVM, which means
the VMM is in charge of setting each subleaf starting from index 2
up to index N, depending on the number of EPC sections.
These subleaves 2 through N are not listed as part of the supported
CPUID entries from KVM. But it's important to set them as long as index
0 and 1 are present and indicate that SGX is supported.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Instead of passing the GuestMemoryMmap directly to the CpuManager upon
its creation, it's better to pass a reference to the MemoryManager. This
way we will be able to know if SGX EPC region along with one or multiple
sections are present.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The SGX EPC region must be exposed through the ACPI tables so that the
guest can detect its presence. The guest only get the full range from
ACPI, as the specific EPC sections are directly described through the
CPUID of each vCPU.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the presence of one or multiple SGX EPC sections from the VM
configuration, the MemoryManager will allocate a contiguous block of
guest address space to hold the entire EPC region. Within this EPC
region, each EPC section is memory mapped.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Introducing the new CLI option --sgx-epc along with the OpenAPI
structure SgxEpcConfig, so that a user can now enable one or multiple
SGX Enclave Page Cache sections within a contiguous region from the
guest address space.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Extract the code that is used by vhost_user_block from the
virtio-devices crate to remove the dependencies on unrequired
functionality such as the virtio transports.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This commit adds required environment configurations to the
`dev_cli.sh` and a Jenkins stage to enable AArch64 binary
building using musl toolchain.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
As the current AArch64 integration test builds kernel every time,
which would take unnecessary time in CI and therefore not ideal.
This commit refactors the AArch64 kernel building strategy to:
1. Keep the Linux kernel source code directory instead of deleting
it everytime after the kernel is built.
2. In integration test script, check if the Linux kernel source
code directory exists. If so use `git fetch && git checkout -f` to
keep the source code always updated, else run `git clone` to get
the source code.
3. Copy config file in and then compile the kernel.
Fixes: https://github.com/cloud-hypervisor/cloud-hypervisor/issues/1444
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This simplies some of the handling for PCI BARs particularly with
respect to snapshot and restore. No attempt has been made to handle the
64-bit bar handling in a different manner to that which was used before.
Fixes: #1153
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In this commit we saved the BDF of a PCI device and set it to "devid"
in GSI routing entry, because this field is mandatory for GICv3-ITS.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
The type is now hypervisor::Vm. Switch from KVM specific name vm_fd to a
generic name just like 8186a8eee6 ("vmm: interrupt: Rename vm_fd").
No functional change.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Under high load, the VM might take some time to hotplug the disk after
the hotplug command has been issued. For this reason, let's put a 10s
sleep before checking for the presence of the new disk.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We want to give the time to the nested VM to be fully ready before we
check it's correctly setup. This involves 3 layers of virtualization
when running the CI on Azure, which added to the high load happening
because of the parallelization, adds up to the start up time.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The virtio configuration structures have been slightly modified between
5.6-rc4 and 5.8-rc4, forcing the virtio-iommu device to be updated
accordingly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Updating the kernel from 5.6-rc4 to 5.8-rc4 allows us to remove the
dependency on both virtio-vsock and virtio-mem patches as they are now
part of the upstream kernel. We're still carrying virtio-iommu and
virtio-fs patches.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Move the definition of RawFile from virtio-devices crate into qcow
crate. All the code that consumes RawFile also already depends on the
qcow crate for image file type detection so this change breaks the
need for the qcow crate to depend on the very large virtio-devices
crate.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
As compiling without acpi (implied by mmio) means that the VM will
terminate on i8042 reset we cannot test the reboot.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Now that vhost_user_net crate does not depend on the virtio-devices
crate it is no longer compiled differently based on the mmio or pci
features.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Move NetQueuePair and the related NetCounters into the net_util crate.
This means that the vhost_user_net crate now no longer depends on
virtio-devices and so does not depend on the pci, qcow or other similar
crates. This significantly simplifies the build chain for this backend.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
By moving the code for opening the two RX and TX queues into a shared
location we are starting to remove the requirement for the
vhost-user-net backend to depend on the virtio-devices crate which in of
itself depends on many other crates that are not necessary for the
backend to function.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Rather than use an embedded String inside the MultiQueueSupport error
value use two different values to differentiate the two cases.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
By moving the code for opening the TAP device into a shared location we
are starting to remove the requirement for the vhost-user-net backend to
depend on the virtio-devices crate which in of itself depends on many
other crates that are not necessary for the backend to function.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
It gets bubbled all the way up from hypervsior crate to top-level
Cargo.toml.
Cloud Hypervisor can't function without KVM at this point, so make it
a default feature.
Fix all scripts that use --no-default-features.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Check if increasing the time after the VM is spawned help with getting
more stable numbers for the base PSS.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit store balloon size to MemoryConfig.
After reboot, virtio-balloon can use this size to inflate back to
the size before reboot.
Signed-off-by: Hui Zhu <teawater@antfin.com>
The implementation of the virtio-balloon was slightly wrong as it was
generating the GPA (Guest Physical Address) from the PFN (Page Frame
Number) which was a u32. That means the GPA was created as a u32, and
later a cast was done to extend it to a u64 type. Unfortunately, by
doing so, the GPA was wrong if the value was supposedly more than 32
bits.
That's why the PFN is casted into a u64 before the GPA is generated,
which creates the GPA on 64 bits directly.
Additionally, this patch simplifies the process_queue() function,
relying on multiple vm-memory helpers.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit adds a Jenkins stage for AArch64 integration test,
and the test is carried out using the GNU toolchain.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
The compiled AArch64 linux kernel by running `make` is in PE format
instead of vmlinux, vmlinux.pvh and bzImage format. Therefore we
need to add integration test for this case.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This commit adds supporting components and code for enabling the
AArch64 integration tests, including:
1. A Linux kernel config file to build kernel on AArch64 machines.
2. Refactoring the `run_integration_test.sh` to architecture
specific scripts for readability.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Remove the vmm dependency from vhost_user_block and vhost_user_net where
it was existing to use config::OptionParser. By moving the OptionParser
to its own crate at the top-level we can remove the very heavy
dependency that these vhost-user backends had.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Refactored the construction of KVM IOCTL rules for Seccomp.
Separating the rules by architecture can reduce the risk of bugs and
attacks.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
In order to move the hypervisor specific parts of the VM exit handling
path, we're defining a generic, hypervisor agnostic VM exit enum.
This is what the hypervisor's Vcpu run() call should return when the VM
exit can not be completely handled through the hypervisor specific bits.
For KVM based hypervisors, this means directly forwarding the IO related
exits back to the VMM itself. For other hypervisors that e.g. rely on the
VMM to decode and emulate instructions, this means the decoding itself
would happen in the hypervisor crate exclusively, and the rest of the VM
exit handling would be handled through the VMM device model implementation.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Fix test_vm unit test by using the new abstraction and dropping some
dead code.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
The code is purely for maintaining an internal counter. It is not really
tied to KVM.
No functional change.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
There are several dependencies that need updating so update them
manually rather than relying on dependabot. This will reduce the load on
the CI.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The binary is still built in the same location but the source code and
the dependencies for it come from the vhost_user_block crate itself.
The binary will be built with:
`cargo build --all --bin vhost_user_block` or just `cargo build --all`
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The binary is still built in the same location but the source code and
the dependencies for it come from the vhost_user_net crate itself.
The binary will be built with:
`cargo build --all --bin vhost_user_net` or just `cargo build --all`
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
With vhost_user_fs binary moved to its own crate the dependencies in the
top level can be trimmed significantly.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The binary is still built in the same location but the source code and
the dependencies for it come from the vhost_user_fs crate itself.
The binary will be built with:
`cargo build --all --bin vhost_user_fs` or just `cargo build --all`
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In preparation for splitting the binaries into their own crates start
building all the binaries in the workspace as part of the integration
testing suite.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In preparation for splitting the binaries into their own crates start
building all the binaries in the workspace when running the build
command inside the container.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In preparation for splitting the binaries into their own crates start
building all the binaries in the workspace when doing a build as part of
the GitHub actions.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The _fd suffix is KVM specific. But since it now point to an hypervisor
agnostic hypervisor::Vm implementation, we should just rename it vm.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The _fd suffix is KVM specific. But since it now point to an hypervisor
agnostic hypervisor::Vm implementation, we should just rename it vm.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The fd naming is quite KVM specific. Since we're now using the
hypervisor crate abstractions, we can rename those into something more
readable and meaningful.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The fd naming is quite KVM specific. Since we're now using the
hypervisor crate abstractions, we can rename those into something more
readable and meaningful.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The fd naming is quite KVM specific. Since we're now using the
hypervisor crate abstractions, we can rename those into something more
readable and meaningful. Like e.g. vcpu or vm.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Now that our CI has transitioned from ClearLinux to Ubuntu images
exclusively, let's update the documentation to refer to Ubuntu images
instead of ClearLinux's ones.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that the project relies on Ubuntu images, the documentation related
to the creation of a custom image has been updated. It's important to
note this procedure could be applied to any other distribution, given
that the package manager's commands would be slightly different.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to differentiate tests that can be run in parallel versus
tests that must be run on their own, we move all tests into dedicated
modules.
The point is to avoid glitches in results that can be caused by the fact
that other tests (hence VMs) are running at the same time.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Split the generic virtio code (queues and device type) from the
VirtioDevice trait, transport and device implementations.
This also simplifies the feature handling in vhost_user_backend as the
vm-virtio crate is no longer has any features.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Move the CI to rely entirely on Ubuntu cloud images. It's worth noting
that both QCOW2 and RAW images from Ubuntu Focal Fossa have been
modified to include the tools needed from integration tests.
This means fio, iperf, iperf3, netcat and socat have been added to the
image. The snapd package have been fully removed as it was expecting the
support for squashfs (not present when using our own kernel from direct
kernel boot), which was causing some failures, and was preventing
cloud-init from terminating properly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Updating the cloud-init files related to Ubuntu images is needed to be
able to use Ubuntu images for running all integration tests.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
With QCOW disk images the space needed is greater than the size of the
iamge as any "zero" blocks in the image are allocated when they are
touched making the image bigger.
Here we add a threshold of 6GiB with added debugging messages to ensure
that there is sufficient disk space to run the tests.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This update of Dockerfile will add support to the AArch64
integration tests and musl building, including:
1. Installation of the missing `setcap` binary on AArch64.
On AArch64, `setcap` binary should be installed via `libcap2-bin`.
This binary is needed in the integration test.
2. Related support for the cloud-hypervisor binary building by
musl toolchain.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Extend the existing integration test test_snapshot_restore by testing
with more than one vCPU.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
On x86 architecture, we need to save a list of MSRs as part of the vCPU
state. By providing the full list of MSRs supported by KVM, this patch
fixes the remaining snapshot/restore issues, as the vCPU is restored
with all its previous states.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Add a new function to the hypervisor trait so that the caller can
retrieve the list of MSRs supported by this hypervisor.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Some vCPU states such as MP_STATE can be modified while retrieving
other states. For this reason, it's important to follow a specific
order that will ensure a state won't be modified after it has been
saved. Comments about ordering requirements have been copied over
from Firecracker commit 57f4c7ca14a31c5536f188cacb669d2cad32b9ca.
This patch also set the previously saved VCPU_EVENTS, as this was
missing from the restore codepath.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The logic can be shared among hypervisor implementations.
The 'static bound is used such that we don't need to deal with extra
lifetime parameter everywhere. It should be okay because we know the
entry type E doesn't contain any reference.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
This trait contains a function which produces a interrupt routing entry.
Implement that trait for KvmRoutingEntry and rewrite the update
function.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
The observation is only the route entry is hypervisor dependent.
Keep a definition of KvmMsiInterruptManager to avoid too much code
churn.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
The observation is that only the route field is hypervisor specific.
Provide a new function in blanket implementation. Also redefine
KvmRoutingEntry with RoutingEntry to avoid code churn.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
The observation is that the GSI hashmap remains untouched before getting
passed into the MSI interrupt manager. We can create that hashmap
directly in the interrupt manager's new function.
The drops one import from the interrupt module.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
The OVMF firmware loops around looking for an entry marking the end of
the table. Without this entry processing the tables is an infinite loop.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Taken from crosvm: 44336b913126d73f9f8d6854f57aac92b5db809e and adapted
for Cloud Hypervisor.
This is basic and incomplete support but Linux correctly finds the DMI
data based on this:
root@clr-c6ed47bc1c9d473d9a3a8bddc50ee4cb ~ # dmesg | grep -i dmi
[ 0.000000] DMI: Cloud Hypervisor cloud-hypervisor, BIOS 0
root@clr-c6ed47bc1c9d473d9a3a8bddc50ee4cb ~ # dmesg | grep -i smbio
[ 0.000000] SMBIOS 3.2.0 present.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Update the API documentation to reflect that the hotplug APIs return
data about the device as well as the newly added /vm.counters API.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This is a hash table of string to hash tables of u64s. In JSON these
hash tables are object types.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Initially the licensing was just Apache-2.0. This patch changes
the licensing to dual license Apache-2.0 OR BSD-3-Clause
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Each virtio thread was reading/draining the pause_evt pipe when
detecting the associated event. Problem is, when a virtio device has
multiple threads, they all share the same pause_evt pipe, which can
prevent some threads from receiving the event. If the first thread to
catch the event is quickly clearing the pipe, some other threads might
simply miss the event and they will not enter the "paused" state as
expected.
This is a behavior that was spotted with virtio-net as it usually uses
2 threads by default (1 for TX/RX queues and 1 for the control queue).
The way to solve this issue is by letting each thread drain the pipe
during the resume codepath, that is after the thread has been unparked.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Because we don't want the guest to miss any event triggered by the
emulation of devices, it is important to resume all vCPUs before we can
resume the DeviceManager with all its associated devices.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We need consistency between pause/resume and snapshot/restore
operations. The symmetrical behavior of pausing/snapshotting
and restoring/resuming has been introduced recently, and we must
now ensure that no matter if we're using pause/resume or
snapshot/restore features, the resulting VM should be running in
the exact same way.
That's why the vCPU state is now stored upon VM pausing. The snapshot
operation being a simple serialization of the previously saved state.
The same way, the vCPU state is now restored upon VM resuming. The
restore operation being a simple deserialization of the previously
restored state.
It's interesting to note that this patch ensures time consistency from a
guest perspective, no matter which clocksource is being used. From a
previous patch, the KVM clock was saved/restored upon VM pause/resume.
We now have the same behavior for TSC, as the TSC from the vCPUs are
saved/restored upon VM pause/resume too.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Instead of calling the resume() function from the CpuManager, which
involves more than what is needed from the shutdown codepath, and
potentially ends up with a deadlock, we replace it with a subset.
The full resume operation is reserved for a VM that has been paused.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We want each Vcpu to store the vCPU state upon VM pausing. This is the
reason why we need to explicitly implement the Pausable trait for the
Vcpu structure.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
When set_user_memory_region was moved to hypervisor crate, it was turned
into a safe function that wrapped around an unsafe call. All but one
call site had the safety statements removed. But safety statement was
not moved inside the wrapper function.
Add the safety statement back to help reasoning in the future. Also
remove that one last instance where the safety statement is not needed .
No functional change.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
That removes one more KVM-ism in VMM crate.
Note that there are more KVM specific code in those files to be split
out, but we're not at that stage yet.
No functional change.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Add a simple test to check that the data from the counters matches what
is expected and that the value of the counters increases after an
operation that will hit all counters.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Collate the virtio device counters in DeviceManager for each device that
exposes any and expose it through the recently added HTTP API.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The counters are a hash of device name to hash of counter name to u64
value. Currently the API is only implemented with a stub that returns an
empty set of counters.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Through the counters() function on the trait expose the accumulated
counters.
TEST=Observe that the counters from the VM match those from the tap on
the host (RX-TX inverted) and inside the guest (non inverted.)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The counters are a hash of counter name to (wrapping) u64 value. The
interpretation layer is responsible for converting this data into a
rate.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This corresponds to QEMU's 63659fe74e76f5c52854 commit.
the setattr code uses both fchmod and fchmodat in different cases,
however we only had fchmodat in the whitelist.
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
The structure is tightly coupled with KVM. It uses KVM specific
structures and calls. Add Kvm prefix to it.
Microsoft hypervisor will implement its own interrupt group(s) later.
No functional change intended.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Now that the VMM uses KVM_KVMCLOCK_CTRL from the KVM API, it must be
added to the seccomp filters list.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Through the newly added API notify_guest_clock_paused(), this patch
improves the vCPU pause operation by letting the guest know that each
vCPU is being paused. This is important to avoid soft lockups detection
from the guest that could happen because the VM has been paused for more
than 20 seconds.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
It's important that on restore path, the CpuManager's vCPU gets filled
with each new vCPU that is being created. In order to cover both boot
and restore paths, the list is being filled from the common function
create_vcpu().
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that the VMM uses both KVM_GET_CLOCK and KVM_SET_CLOCK from the KVM
API, they must be added to the seccomp filters list.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to maintain correct time when doing pause/resume and
snapshot/restore operations, this patch stores the clock value
on pause, and restore it on resume. Because snapshot/restore
expects a VM to be paused before the snapshot and paused after
the restore, this covers the migration use case too.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
When a request is made to increase the number of vCPUs in the VM attempt
to reuse any previously removed (and hence inactive) vCPUs before
creating new ones.
This ensures that the APIC ID is not reused for a different KVM vCPU
(which is not allowed) and that the APIC IDs are also sequential.
The two key changes to support this are:
* Clearing the "kill" bit on the old vCPU state so that it does not
immediately exit upon thread recreation.
* Using the length of the vcpus vector (the number of allocated vcpus)
rather than the number of active vCPUs (.present_vcpus()) to determine
how many should be created.
This change also introduced some new info!() debugging on the vCPU
creation/removal path to aid further development in the future.
TEST=Expanded test_cpu_hotplug test.
Fixes: #1338
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
After the vCPU has been ejected and the thread shutdown it is useful to
clear the "kill" flag so that if the vCPU is reused it does not
immediately exit upon thread recreation.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
These messages are intended to be useful to support debugging related to
vCPU hotplug/unplug issues.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Now that snapshot/restore is symmetrical, that is the VM must be paused
before it is snapshot and it must be resumed after it's been restored,
the integration test is updated accordingly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The same way the VM and the vCPUs are restored in a paused state, all
devices associated with the device manager must be restored in the same
paused state.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Because we need to pause the VM before it is snapshot, it should be
restored in a paused state to keep the sequence symmetrical. That's the
reason why the state machine regarding the valid VM's state transition
needed to be updated accordingly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
To follow a symmetrical model, and avoid potential race conditions, it's
important to restore a previously snapshot VM in a "paused" state.
The snapshot operation being valid only if the VM has been previously
paused.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
When the hypervisor crate was introduced, a few places that handled
errors were commented out in favor of unwrap, but that's bad practice.
Restore proper error handling in those places in this patch.
We cannot use from_raw_os_error anymore because it is wrapped deep under
hypervisor crate. Create new custom errors instead.
Fixes: e4dee57e81 ("arch, pci, vmm: Initial switch to the hypervisor crate")
Signed-off-by: Wei Liu <liuwe@microsoft.com>
This commit fixes some warnings introduced in the previous
hyperviosr crate PR.Removed some unused variables from arch/aarch64
module.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Start moving the vmm, arch and pci crates to being hypervisor agnostic
by using the hypervisor trait and abstractions. This is not a complete
switch and there are still some remaining KVM dependencies.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Implement the vCPU state getter and setter separately from the initial
KVM Hypervisor trait implementation, mostly for readability purposes.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
For each of the traits we are defining kvm related structures
and add the trait implementation to the structs. For more information
please see the kvm-ioctls and kvm-bindings crate.
This is a standalone implementation that does not include the switch of
the Cloud-Hypervisor vmm and arch crates to it.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
As the only hypervisor that Cloud-Hypervisor supports is KVM, the
Hypervisor trait accomodates for the upcoming KVM implementation.
This trait will be instanciated at build time through hypervisor
specific features, i.e. it's not aiming at run-time selection of
hypervisors for Cloud-Hypervisor.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
This VM trait should be implemented by each underlying hypervisor.
Previously created hypervisor object should create the VM based on
already selected hypervisor. This is just the trait definition. For each
of supported hypervisor we need to implement the trait. Later we will
implement this trait for KVM and then Microsoft Hyper-V.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
This Vcpu trait should be implemented by each underlying hypervisor.
Previously created hypervisor object should create the VM based on
already selected hypervisor and Vm object should create this vcpu
object based on same hyperviosr. Each of this object should be
referenced by trait object i.e <dyn Vcpu>.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The purpose of this trait is to add support for other hypervisors than
KVM, like e.g. Microsoft Hyper-V.
Further commits will define additional hypervisor related traits like
Vcpu and Vm. Each of the supported hypervisor will need to implement all
traits defined from the hypervisor crate.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Add multiple integration tests with various different CPU topologies and
check that they work as expected.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
There are two CPUID leaves for handling CPU topology, 0xb and 0x1f. The
difference between the two is that the 0x1f leaf (Extended Topology
Leaf) supports exposing multiple die packages.
Fixes: #1284
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The extended topology leaf (0x1f) also needs to have the APIC ID (which
is the KVM cpu ID) set. This mirrors the APIC ID set on the 0xb topology
leaf
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Rather than saving the individual parts into the CpuManager save the
full struct as it now also contains the topology data.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This needed to be updated to include specifying the boot and maxmium
vCPUs as well as the newly added topology for those vCPUs.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This allows the user to optionally specify the desired CPU topology. All
parts of the topology must be specified and the product of all parts
must match the maximum vCPUs.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This patch has been cherry-picked from the Firecracker tree. The
reference commit is 78ca0a942f32140465c67ea4b45d68c52c72d751.
Signed-off-by: Gabriel Ionescu <gbi@amazon.com>
Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
This patch has been cherry-picked from the Firecracker tree. The
reference commit is 6dbe8e021a64ba3742081741a7538cdfd93a102e.
Signed-off-by: Gabriel Ionescu <gbi@amazon.com>
Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
This patch has been cherry-picked from the Firecracker tree. The
reference commit is 78819f35f63f5777a58e3e1e774b3270b32881ed.
The vsock TX buffer flush operation would report inconsistent results,
under specific circumstances.
The flush operation is performed in two steps, since it's dealing with a
ring buffer, an the data to be flushed may wrap around. If the first
step was successful, but the second one failed, the whole flush
operation would report an error, thus causing flow control accounting to
lose track of the bytes that were successfully written by the first
pass.
This commit changes the flush behavior to always report success when
some data has been written to the backing stream.
Signed-off-by: Dan Horobeanu <dhr@amazon.com>
Signed-off-by: Gabriel Ionescu <gbi@amazon.com>
Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
This patch has been cherry-picked from the Firecracker tree. The
reference commit is 2da612a9cdce85c91fb54ab22d950ec6ccc93b27.
Fixed a bug introduced by a271d08f0b1ba0ee82761cd49244b6a8017bcede,
whereby the flow control accouting would be off by a few bytes, for
host-initiated connections.
The connection ack message ("OK <port_num><CR>") was accounted for as
data sent by the guest, so its length was substracted from the total
amount of data the guest was allowed to send.
This commit changes the way this ack message is sent, so that it
bypasses flow control accouting.
Signed-off-by: Dan Horobeanu <dhr@amazon.com>
Signed-off-by: Gabriel Ionescu <gbi@amazon.com>
Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
This patch has been cherry-picked from the Firecracker tree. The
reference commit is 109e631566350867dafa4b16c3919dfd1533eeea.
This commit changes the vsock connection state machine behavior to absorb
any EWOULDBLOCK errors recevied while handling an EPOLLOUT event. Previously,
this condition would lead to immediate connection termination.
Signed-off-by: Dan Horobeanu <dhr@amazon.com>
Signed-off-by: Gabriel Ionescu <gbi@amazon.com>
Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
This patch has been cherry-picked from the Firecracker tree. The
reference commit is 660d18cf7fee5b38c3b1b17a5da6544b9025909d.
Apparently, epoll_wait sometimes yields false EPOLLIN events (i.e. events
follwing which read() would fail with EWOULDBLOCK). This would cause the
vsock connection state machine to terminate connections, since an error
was detected on the underlying Unix socket.
This commit changes the vsock connection state machine code to handle such
erroneous EPOLLIN events by absorbing EWOULDBLOCK read() errors.
Signed-off-by: Dan Horobeanu <dhr@amazon.com>
Signed-off-by: Gabriel Ionescu <gbi@amazon.com>
Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
This patch has been cherry-picked from the Firecracker tree. The
reference commit is 1cc8b8a678eb28b20f5843556bdb7fbb2dfa6284.
Fixed a logical error in the vsock flow control, that would cause credit
update packets to not be sent at the right time.
Signed-off-by: Dan Horobeanu <dhr@amazon.com>
Signed-off-by: Gabriel Ionescu <gbi@amazon.com>
Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
This patch has been cherry-picked from the Firecracker tree. The
reference commit is d2475773557c82d2abad2fc8bdf69e7d01444109.
Fixed a vsock muxer issue that would cause a connection to be removed
from the RX queue, even though it still had pending RX data.
Signed-off-by: Dan Horobeanu <dhr@amazon.com>
Signed-off-by: Gabriel Ionescu <gbi@amazon.com>
Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
This patch adds `is_empty` method to VsockPacket to fix the
following clippy error:
error: item `vsock::packet::VsockPacket` has a public `len` method but no corresponding `is_empty` method
--> vm-virtio/src/vsock/packet.rs:100:1
|
100 | / impl VsockPacket {
101 | | /// Create the packet wrapper from a TX virtq chain head.
102 | | ///
103 | | /// The chain head is expected to hold valid packet header data. A following packet buffer
... |
334 | | }
335 | | }
| |_^
|
= note: `-D clippy::len-without-is-empty` implied by `-D warnings`
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#len_without_is_empty
Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
Currently, not every feature of the cloud-hypervisor is enabled
on AArch64, which means that on AArch64 machines, the
`run_unit_tests.sh` needs to be tailored and some unit test cases
should be run on x86_64 only.
Also this commit fixes the typo and unifies `Arm64` and `AArch64`
in the AArch64 document.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Its test case calls remove unconditionally. Instead of making the test
code call remove conditionally, removing the pci_support dependency
simplifies things -- that function is just a wrapper around HashMap's
remove function anyway.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
cloud-hypervisor: 763.978581807s: ERROR:pci/src/vfio.rs:651 -- failed to remove all guest memory regions from iommu table
when poweroff a vm with vfio device, clh will finally remove all guest memory region from iommu table
with the method unset_dma_map, not method setup_dma_map.
Signed-off-by: LiYa'nan <oliverliyn@gmail.com>
Upon PCI hotplug, the VMM now returns some information about the device
name and the associated b/d/f. This patch extends the integration tests
so that we validate the response is the one that is expected.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that PCI device hotplug returns a response, the OpenAPI definition
must reflect it, describing what is expected to be received.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to provide a more comprehensive b/d/f to the user, the
serialization of PciDeviceInfo is implemented manually to control the
formatting.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This patch completes the series by connecting the dots between the HTTP
frontend and the device manager backend.
Any request to hotplug a VFIO, disk, fs, pmem, net, or vsock device will
now return a response including the device name and the place of the
device in the PCI topology.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Pass from the device manager to the calling code the information about
the PCI device that has just been hotplugged.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to provide the device name and PCI b/d/f associated with a
freshly hotplugged device, the hotplugging functions from the device
manager return a new structure called PciDeviceInfo.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The support of AArch64 is in very early stage. The steps in building and
runing on X86 and AArch64 can not align well yet. Adding AArch64 content
to README.md would produce much divergence.
Adding a guide in docs/ folder could be a better way to start now.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
X86 and AArch64 work in different ways to shutdown a VM.
X86 exit VMM event loop through ACPI device;
AArch64 need to exit from CPU loop of a SystemEvent.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Screened IO bus because it is not for AArch64.
Enabled Serial, RTC and Virtio devices with MMIO transport option.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Implemented GSI allocator and system allocator for AArch64.
Renamed some layout definitions to align more code between architectures.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
This fixes the build warning below
warning: use of deprecated item 'tap::tests::pnet::datalink::NetworkInterface::mac_address': Please use NetworkInterface's field 'mac' instead.
--> net_util/src/tap.rs:530:24
|
530 | (interface.mac_address(), tx, rx)
| ^^^^^^^^^^^
|
= note: `#[warn(deprecated)]` on by default
Signed-off-by: Anatol Belski <ab@php.net>
For correctness, when the CPUID supports the LA57 feature, the VMM sets
the CR4.LA57 register, which means a fifth level of page table might be
needed. Even if it's not needed because the kernel should not use
addresses over 1GiB, it's better to define this new level anyway.
This patch only applies to the Linux boot codepath, which means it
affects both vmlinux without PVH and bzImage binaries. The bzImage
does not need this since the page tables and CR4 registers are set in
the decompression code from the kernel.
And for vmlinux with PVH, if we follow the PVH specification, the kernel
must be responsible for setting things up, but the implementation is
missing. This means for now that PVH does not support LA57 with 5 levels
of paging.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
"cc" is invoked as part of the Cloud Hypervisor Rust build however due
to a copy and paste error the wrong variable was being tested for
overriding the CC and the CFLAGS.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This revised version of the patch reuses the back-off strategy from
'ssh_command()' to deal with varying booting time.
Fixes: #1209
Signed-off-by: Bo Chen <chen.bo@intel.com>
Some CPUs might support up to 52 bits of addressable space, that's the
reason why we need the guest kernel to support 5 levels of page table.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to workaround a Linux bug that happens when we place devices at
the end of the physical address space on recent hardware (52 bits limit)
we reduce the MMIO address space by one 4k page. This way, nothing gets
allocated in the last 4k of the address space, which is negligible given
the amount of space in the address space.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In case the host CPU exposes the support for LA57 feature through its
cpuid, the CR4.LA57 bit is enabled accordingly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Currently released vm-memory uses aligned and volatile copying for all
data. The version in the fork only uses the assured (and slower) path
for data upto the natural data width.
Fixes: #1258
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This commit adds the implementation for the AArch64 PL031
Real Time Clock (RTC) that provides a long time base counter. This
is achieved by generating an interrupt signal after counting a
programmed number of cycles of a real-time clock input. The AArch64
guest VM of the cloud-hypervisor will use this RTC to sync the time
in itself.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
The action of "vm.delete" should not report errors on non-booted
VMs. This patch also revised the "docs/api.md" to reflect the right
'Prerequisites' of different API actions, e.g. on "vm.delete" and
"vm.boot".
Fixes: #1110
Signed-off-by: Bo Chen <chen.bo@intel.com>
This removes the need to use CAP_NET_ADMIN privileges and instead the
host MAC addres is either provided by the user or alternatively it is
retrieved from the kernel.
TEST=Run cloud-hypervisor without CAP_NET_ADMIN permission and a
preconfigured tap device:
sudo ip tuntap add name tap0 mode tap
sudo ifconfig tap0 192.168.249.1 netmask 255.255.255.0 up
cargo clean
cargo build
target/debug/cloud-hypervisor --serial tty --console off --kernel ~/src/rust-hypervisor-firmware/target/target/release/hypervisor-fw --disk path=~/workloads/clear-33190-kvm.img --net tap=tap0
VM was also rebooted to check that works correctly.
Fixes: #1274
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This can be used to preserve the host MAC address as part of the
configuration when the TAP device is precreated.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This allows an existing TAP interface to be used without needing
CAP_NET_ADMIN permissions on the Cloud Hypervisor binary as the ioctl to
bring up the interface is avoided.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
If the server returns an error then print out the response body if there
is one present.
Fixes: #1262
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In AArch64 prototype, there are code to construct the flattened device
tree, and to make such code compilable we need to install libfdt-dev. In
normal situation, this installation process can be done by either installing
libfdt-dev locally or in the development container.
Before formal AArch64 CI is setup, we use the workaround in this commit
to install libfdt in the github cross-build workflow.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
When booting VM on AArch64 machines, we need to construct the
flattened device tree before loading kernel. Hence here we add
the implementation of the flattened device tree for AArch64.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
As on AArch64 systems we need register mpidr to create the
flattened device tree, here in this commit we add ported AArch64
register implementation from Firecracker and related changes to
make this commit build.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This commit adds ported code of Generic Interrupt Controller (GIC)
software implementation for AArch64, including both GICv2 and
GICv3 devices. These GIC devices are actually emulated by the
host kernel through KVM and will be used in the guest VM as the
interrupt controller for AArch64.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This commit adds the memory layout design for AArch64 in
`arch/src/aarch64/layout.rs` and related changes in
`arch/src/lib.rs` to make sure this commit can build.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
The API has change to use generic GuestMemory trait:
pub fn get_host_address_range<M: GuestMemory>(
mem: &M,
addr: GuestAddress,
size: usize,
) -> Option<*mut u8> {
Signed-off-by: Arron Wang <arron.wang@intel.com>
There is a much stronger PCI dependency from vfio_pci.rs than a VFIO one
from pci/src/vfio.rs. It seems more natural to have the PCI specific
VFIO implementation in the PCI crate rather than the other way around.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Now the flow of both architectures are aligned to:
1. load kernel
2. create VCPU's
3. configure system
4. start VCPU's
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Between X86 and AArch64, there is some difference in booting a VM:
- X86_64 can setup IOAPIC before creating any VCPU.
- AArch64 have to create VCPU's before creating GIC.
The old process is:
1. load_kernel()
load kernel binary
configure system
2. activate_vcpus()
create & start VCPU's
So we need to separate "activate_vcpus" into "create_vcpus" and
"activate_vcpus" (to start vcpus only). Setup GIC and create FDT
between the 2 steps.
The new procedure is:
1. load_kernel()
load kernel binary
(X86_64) configure system
2. create VCPU's
3. (AArch64) setup GIC
4. (AArch64) configure system
5. start VCPU's
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
The general handling of VIRTIO_RING_F_EVENT_IDX is in the
vhost_user_backend functionality and the net specific handling is in the
NetQueuePair from virtio-net.
As such enabling for the vhost-user-net backend is just the case of
adding the feature.
Fixes: #789
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
If VIRTIO_RING_F_EVENT_IDX is negotiated only generate suppress
interrupts if the guest has asked us to do so.
Fixes: #788
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In some situations it is seen that the first interrupt sent to the guest
is lost upon a restore (due to the tap worker being awake ahead of the
vPUs).
This causes problems with VIRTIO_RING_F_EVENT_IDX interrupt suppression
as the guest will not be interrupted again in order to mitigate this we
always interrupt the guest until the device itself has been signalled by
the guest.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The logic for handling the networking queues can now be shared between
the version running in vhost-user-net and vm-virtio.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This requires exposing the struct members and also using Option<..>
types for the main epoll fd and the memory as they are initialised later
in vhost-user-net.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Split handling of behaviour that is independent of the device itself so
that it can be reused in the vhost-user-net device.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Split out functions that work just on the TAP device and queues. Whilst
doing so also improve the error handling to return Results rather than
drop errors.
This change also addresses a bug where the TAP event suppression could
ineffectual because it was being enabled immediately after it may have
been disabled:
resume_rx -> rx_single_frame -> unregister_listener -> resume_rx ->
register_listener.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Instead of responding only headers with error code, we now return
complete error responses to HTTP requests with errors (e.g. undefined
endpoints and InternalSeverError).
Fixes: #472
Signed-off-by: Bo Chen <chen.bo@intel.com>
When doing self spawning the child will attempt to set the umask() again. Let
it through the seccomp rules so long as it the safe mask again.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This commit only implements the InterruptController crate on AArch64.
The device specific part for GIC is to be added.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
IOAPIC, a X86 specific interrupt controller, is referenced by device
manager and CPU manager. To work with more architectures, a common
type for all architectures is needed.
This commit introduces trait InterruptController to provide architecture
agnostic functions. Device manager and CPU manager can use it without
caring what the underlying device is.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
The result of the workflow can be seen in Checks tab of a PR.
Two targets have been added:
- stable aarch64-unknown-linux-gnu
- stable aarch64-unknown-linux-musl
Note: a temporary step was added before building. We used "sed" command
to remove "with-serde" feature of kvm-bindings in vmm/Cargo.toml.
This step should be removed in future when kvm-bindings is ready.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
This is a preparing commit to build and test CH on AArch64. All building
issues were fixed, but no functionality was introduced.
For X86, the logic of code was not changed at all.
For ARM, the architecture specific part is still empty. And we applied
some tricks to workaround lint warnings. But such code will be replaced
later by other commits with real functionality.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Updated Dockerfile to work with multiple architectures.
Updated dev_cli.sh to:
1. Build container image before AArch64 image is ready in public.
2. Adjust default feature collection on AArch64.
3. Workaround a build problem with musl on AArch64.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
This config option provided very little value and instead we now enable
this feature (which then lets the guest control the cache mode)
unconditionally.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Correctly implement the virtio specification by setting the writeback
field on the request based on the algorithm in the spec.
TEST=Boot with hypervisor-firmware with CH in verbose mode. See info
level messages saying cache mode is writethrough in firmware (no support
for flush or WCE). Once in the Linux kernel see messages that mode is
writeback.
Fixes: #1216Fixes: #680
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Correctly implement the virtio specification by setting the writeback
field on the request based on the algorithm in the spec.
TEST=Boot with hypervisor-firmware with CH in verbose mode. See info
level messages saying cache mode is writethrough in firmware (no support
for flush or WCE). Once in the Linux kernel see messages that mode is
writeback.
Fixes: #1216
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When this is set to false the write needs to be followed by a flush on
the underlying disk (leading to a fsync()).
The default behaviour is not changed with this change.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Previous to adding a a trait method to inform the backends of the acked
features backends can use features than the guest has not enabled which
could lead to unpredictable results.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Move the method that is used to decide whether the guest should be
signalled into the Queue implementation from vm-virtio. This removes
duplicated code between vhost_user_backend and the vm-virtio block
implementation.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The implementation of this virtio block (and vhost-user block) command
called a function that was a no-op on Linux. Use the same function as
virtio-pmem to ensure that data is not lost when the guest asks for it
to be flused to disk.
Fixes: #399
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Now that snapshot/restore support has been enabled for virtio-vsock, the
corresponding integration test is expanded with some validation that
virtio-vsock supports to be snapshot and restored.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
When compiled with pci feature, the integration test now validates that
/dev/vdb can be correctly read while being placed behing a virtual
IOMMU.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
For both virtio-mmio and virtio-pci transport layers, we were setting
every field from the saved snapshot during a restore. This is a problem
when we don't want to override specific fields such as iommu_mapping_cb
because the saved snapshot doesn't contain the appropriate information.
That's why this commit sets only the appropriate field from the saved
snapshot during a restore.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Provide implementation for both snapshot() and restore() methods from
the Snapshottable trait, so that we can snapshot and restore a VM with
devices attached to a virtual IOMMU.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The integration test validating that --serial off works correctly was
not properly written as it was using the FW, which by default would use
the kernel command line found in the EFI partition. Unfortunately, this
kernel command line was including "console=ttyS0", which causes the
kernel to try to write to the serial port, even if there's no serial
port being emulated.
The problem is, when no emulation of the serial port is provided, the
default value returned on 0x3f8 is 0, which makes the guest kernel think
that some data needs to be read.
The only way to avoid all this is by ensuring we can control the kernel
command line by removing any occurence of "console=ttyS0" from it.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Explicit call to 'close()' is required on file descriptors allocated
from 'epoll::create()', which is missing for the 'EpollContext' and
'VringWorker'. This patch enforces to close the file descriptors by
reusing the Drop trait of the 'File' struct.
Signed-off-by: Bo Chen <chen.bo@intel.com>
According to the virtio spec the guest should always be interrupted when
"used" descriptors are returned from the device to the driver. However
this was not the case for the TX queue in either the virtio-net
implementation or the vhost-user-net implementation.
This would have meant that the guest could end up with a reduced TX
throughput as it would not know that the packets had been dispatched via
the VMM.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
As the parsing code is reused the flush feature is already implemented
and ready to be used.
Fixes: #1197
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add missing syscall used by the musl build.
TEST=scripts/dev_cli.sh tests --libc musl --integration -- vhost_user_fs_daemon
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Extend the set of tests we have for virtio-net and vhost-user-net to
check for host MAC address setting.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add a new "host_mac" parameter to "--net" and "--net-backend" and use
this to set the MAC address on the tap interface. If no address is given
one is randomly assigned and is stored in the config.
Support for vhost-user-net self spawning was also included.
Fixes: #1177
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Do this by reading the HW address information and then modifying the
HW address to match the desired address. Preserving the rest of the
state including the address type.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The clock_nanosleep system call needs to be whitelisted since the commit
12e00c0f45 introduced the use of a sleep()
function. Without this patch, we can see an error when the VM is paused
or killed.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Implement seccomp; we use one filter for all threads.
The syscall list comes from the C daemon with syscalls added
as I hit them.
The default behaviour is to kill the process, this normally gets
audit logged.
--seccomp none disables seccomp
log Just logs violations but doesn't stop it
trap causes a signal to be be sent that can be trapped.
If you suspect you're hitting a seccomp action then you can
check the audit log; you could also switch to running with 'log'
to collect a bunch of calls to report.
To see where the syscalls are coming from use 'trap' with a debugger
or coredump to backtrace it.
This can be improved for some syscalls to restrict the parameters
to some syscalls to make them more restrictive.
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Like the actions that don't take data such as "pause" or "resume" use a
common handler implementation to remove duplicated code for handling
simple endpoints like the hotplug ones.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Many of the API requests take a similar form with a single data item
(i.e. config for a device hotplug) expand the VmAction enum to handle
those actions and a single function to dispatch those API events.
For now port the existing helper functions to use this new API. In the
future the HTTP layer can create the VmAction directly avoiding the
extra layer of indirection.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Rather than save the save a function pointer and use that instead the
underlying action. This is useful for two reasons:
1. We can ensure that we generate HttpErrors in the same way as the
other endpoints where API error variant should be determined by the
request being made not the underlying error.
2. It can be extended to handle other generic actions where the function
prototype differs slightly.
As result of this refactoring it was found that the "vm.delete" endpoint
was not connected so address that issue.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Extend the EndpointHandler trait to include automatic support for
handling PUT requests. This will allow the removal of lots of duplicated
code in the following commit from the API handling code.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Implement support for setting up a sandbox for running the
service. The technique for this has been borrowed from virtiofsd, and
consists on switching to new PID, mount and network namespaces, and
then switching root to the directory to be shared.
Future patches will implement additional hardening features like
dropping capabilities and seccomp filters.
Signed-off-by: Sergio Lopez <slp@redhat.com>
Changes is vhost crate require VhostUserDaemon users to create and
provide a vhost::Listener in advance. This allows us to adopt
sandboxing strategies in the future, by being able to create the UNIX
socket before switching to a restricted namespace.
Update also the reference to vhost crate in Cargo.lock to point to the
latest commit from the dragonball branch.
Signed-off-by: Sergio Lopez <slp@redhat.com>
Allow callers to provide a file descriptor for /proc/self/fd. This is
useful for sandboxing, as we may be running in a namespace that
doesn't have access to /proc.
Signed-off-by: Sergio Lopez <slp@redhat.com>
Open a file descriptor to /proc/self/fd instead of /proc. We aren't
using any other entries from that directory, and doing this allows us
to keep working even if /proc is no longer present in our
namespace (useful for sandboxing).
Signed-off-by: Sergio Lopez <slp@redhat.com>
There's a simple way to trigger PCI BAR reprogramming for a given
device, by removing it and then hotplugging it back. The Linux kernel
will simply choose to place the BARs at different location than the ones
chosen by Cloud-Hypervisor. By doing so, and creating the snapshot after
this hotplug operation, we can manage to validate that the resource are
correctly restored for a given virtio-pci device.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that snapshot/restore feature is stable for both virtio-mmio and
virtio-pci devices, we re-enable the integration test for validating
snapshot/restore does not get broken.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The ch branch has been rebased to incorporate the latest upstream code
requiring a small change to the unit tests.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Now that virtio-fs has unplug support try unplugging and readding the
virtio-fs device.
Fixes: #1050
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
By passing a reference of the DeviceTree to the AddressManager, we can
now update the DeviceTree whenever a PCI BAR is reprogrammed. This is
mandatory to maintain the correct resources information related to each
virtio-pci device, which will ensure correct information will be stored
upon VM snapshot.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We want to be able to share the same DeviceTree across multiple threads,
particularly to handle the use case where PCI BAR reprogramming might
need to update the tree while from another thread a new device is being
added to the tree.
That's why this patch moves the DeviceTree instance into an Arc<Mutex<>>
so that we can later share a reference of the same mutable tree with the
AddressManager responsible for handling PCI BAR reprogramming.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By using the vector of resources provided by the DeviceNode, the device
manager can store the information related to PCI BARs from a virtio-pci
device. Based on this, and upon VM restoration, the device manager can
restore the BARs in the expected location in the guest address space.
One thing to note is that we only need to provide the VirtioPciDevice
with the configuration BAR (BAR 0) since the SHaredMemory BAR info comes
from the virtio device directly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
From a VirtioPciDevice perspective, there are two types of BARs, either
the virtio configuration BAR or the SHaredMemory BAR.
The SHaredMemory BAR address comes from the virtio device directly as
the memory region had been previously allocated when the virtio device
has been created. So for this BAR, there's nothing to do when restoring
a VM, since the associated virtio device is already restored with the
appropriate resources, hence the BAR will already be at the right
address.
The remaining configuration BAR is different, as we usually get its
address from the SystemAllocator. This means in case we restore a VM,
we must provide this value, bypassing the allocator. This is what this
commit takes care of, by letting the caller set the base address for the
configuration BAR prior to allocating the BARs.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the new field "pci_bdf", a virtio-pci device can be restored at
the same place on the PCI bus it was located before the VM snapshot.
This ensures consistent placement on the PCI bus, based on the stored
information related to each device.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We need a way to store the information about where a PCI device was
placed on the PCI bus before the VM was snapshotted. The way to do this
is by adding an extra field to the DeviceNode structure.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to let the PciBus user choose where a device should be placed
on the bus, a new function get_device_id() is introduced. This will be
helpful in the context of snapshot/restore as the caller will be able to
place the PCI devices on the same slot they were placed before the
snapshot was taken.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Use the ACPI feature to control whether to build the mptable. This is
necessary as the mptable and ACPI RSDP table can easily overwrite each
other leading to it failing to boot.
TEST=Compile with default features and see that --cpus boot=48 now
works, try with --no-default-features --features "pci" and observe the
--cpus boot=48 also continues to work.
Fixes: #1132
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The mmio integration test build currently doesn't use ACPI so piggyback
on that test variation.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The setup_mptables() call which is not used on ACPI builds has a side
effect of testing whether there was enough RAM which one of the unit
tests was relying on. Add a similar check for the RSDP address.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This gives Cloud-Hypervisor the possibility to snapshot and restore a
VM running with virtio-pci devices attached to it.
The VirtioPciDevice snapshot contains a vector of sub-snapshots to store
and restore information related to MsixConfig, VirtioPciCommonConfig and
PciConfiguration structures, along with snapshot data related to
VirtioPciDevice itself.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This structure contains all the virtio generic information, and as part
of restoring a VM with virtio-pci devices, it is important to restore
these values to ensure the device's proper functioning.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The PCI configuration from each PCI device is modified at runtime as we
can expect the guest OS to write to some PCI capability structure, or
move the BAR to a different location in the guest address space.
For all the reasons why such configuration might differ from the initial
configuration, we must store the registers values to be able to restore
them with the right values whenever a PCI device is being restored.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to restore devices relying on MSI-X, the MsixConfig structure
must be restored with the correct values. Additionally, the KVM routes
must be restored so that interrupts can be delivered through KVM the way
they were configured before the snapshot was taken.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Continue to try and track down the instability in the numbers generated
by this test by removing the (unused) network interface. It's possible
we are getting broadcast packets into the tap device and generating a
variable size of allocations.
See: #760
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Switch to using the recently added OptionParser in the code that parses
the block backend.
Fixes: #1092
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Switch to using the recently added OptionParser in the code that parses
the network backend.
Whilst doing this also update the net-backend syntax to use "sock"
rather than socket.
Fixes: #1092
Partially fixes: #1091
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Rather than repeat syntax for the vhost-user-block backend in multiple
places store it in one place and reference it from the required places.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Rather than repeat syntax for the vhost-user-net backend in multiple
places store it in one place and reference it from the required places.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Upon a virtio reset, the driver expects that available and used indexes
will be reset to 0. That's why we need to reset these values from the
VMM for any virtio device that might get reset.
This issue was not detected before because the Vec<Queue> maintained
through VirtioPciDevice or MmioDevice was never updated from the virtio
device thread after the device had been actived. For this reason, upon
reset, both available and used indexes were already at the value 0.
The issue arose when trying to reset a device after the VM was restored.
That's because during the restore, each queue is assigned with the right
available and used indexes before it is passed to the device through the
activate function. And that's why upon reset, each queue was still
assigned with these indexes while it should have been reset to 0.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
All our virtio devices support to be reset, but the virtio-mmio
transport layer was not implemented for it. This patch fixes this
lack of support.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The ch-remote usage says:
OPTIONS:
--api-socket <api-socket> HTTP API socket path (UNIX domain socket).
However it doesn't seem to actually accept that syntax, instead
requiring "--api-socket=". This may be a clap bug however it is resolved
by setting the number of arguments requires to exactly one. Which is
also the actual correct number.
Fixes: #1117Fixes: #1116
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
To aid the debugging of why the test_memory_overhead sometimes fails
print a details of all the named regions that it uses in its
calculation.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The current tests support giving a tap name but they only test with tap
interfaces that already exist.
Fixes: #871
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Stripping the release build for glibc shrinks the size considerably:
$ du -h target/release/cloud-hypervisor
8.5M target/release/cloud-hypervisor
$ strip target/release/cloud-hypervisor
$ du -h target/release/cloud-hypervisor
5.2M target/release/cloud-hypervisor
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
To avoid a race condition where the signal might "miss" the KVM_RUN
ioctl() instead reapeatedly try sending a signal until the vCPU run is
interrupted (as indicated by setting a new per vCPU atomic.)
It important to also clear this atomic when coming out of a paused
state.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
To ensure that the DeviceManager threads (such as those used for virtio
devices) are cleaned up it is necessary to unpark them so that they get
cleanly terminated as part of the shutdown.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
After setting the kill signal flag for the vCPU thread release the pause
flag and unpark the threads. This ensures that that the vCPU thread will
wake up and check the kill signal flag if the VM is paused.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Rather than immediately entering the vCPU run() code check if the kill
signal is set. This allows paused VMs to be shutdown.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This module will be dedicated to DeviceNode and DeviceTree definitions
along with some dedicated unit tests.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This iterator will let the VMM enumerate the resources associated
with the DeviceManager, allowing for introspection.
Moreover, by implementing a double ended iterator, we can get the
hierarchy from the leaves to the root of the tree, which is very
helpful in the context of restoring the devices in the right order.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that the device tree fully replaced the need for a dedicated list of
migratable devices, this commit cleans up the codebase by removing it
from the DeviceManager.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit switches from migratable_devices to device_tree in order to
restore devices exclusively based on the device tree.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit adds an extra field to the DeviceNode so that the structure
can hold a Migratable device. The long term plan is to be able to remove
the dedicated table of migratable devices, but instead rely only on the
device tree.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to hide the complexity chosen for the device tree stored in
the DeviceManager, we introduce a new DeviceTree structure.
For now, this structure is a simple passthrough of a HashMap, but it can
be extended to handle some DeviceTree specific operations.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This device has a dedicated memory region in the guest address space,
which means in case of snapshot/restore, it must be restored in the
exact same location it was during the snapshot.
That's through the resources that we can describe the location of this
extra memory region, allowing the device for correct restoring.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This device has a dedicated memory region in the guest address space,
which means in case of snapshot/restore, it must be restored in the
exact same location it was during the snapshot.
That's through the resources that we can describe the location of this
extra memory region, allowing the device for correct restoring.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the device tree, retrieve the resources associated with a
virtio-mmio device to restore it at the right location in guest address
space. Also, the IRQ number is correctly restored.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Instead of splitting the MMIO allocation and the device creation into
separate functions for virtio-mmio devices, it's is easier to move
everything into the same function as we'll be able to gather resources
in the same place for the same device.
These resources will be stored in the device tree in a follow up patch.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In case the VM is created from scratch, the devices should be created
after the DeviceManager has been created. But this should not affect the
restore codepath, as in this case the devices should be created as part
of the restore() function.
It's necessary to perform this differentiation as the restore must go
through the following steps:
- Create the DeviceManager
- Restore the DeviceManager with the right state
- Create the devices based on the restored DeviceManager's device tree
- Restore each device based on the restored DeviceManager's device tree
That's why this patch leverages the recent split of the DeviceManager's
creation to achieve what's needed.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit performs the split of the DeviceManager's creation into two
separate functions by moving anything related to device's creation after
the DeviceManager structure has been initialized.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the device tree, we now ensure the restore can be done in the
right order, as it will respect the dependencies between nodes.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The DeviceManager itself must be snapshotted in order to store the
information regarding the devices associated with it, which effectively
means we need to store the device tree.
The mechanics to snapshot and restore the DeviceManagerState are added
to the existing snapshot() and restore() implementations.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The DeviceManager now creates a tree of devices in order to store the
resources associated with each device, but also to track dependencies
between devices.
This is a key part for proper introspection, but also to support
snapshot and restore correctly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on rust-vmm code, we copy the Resource structure as it will be
used to define the set of resources associated with a device.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
It's not possible to call UnixListener::Bind() on an existing file so
unlink the created socket when shutting down the Vsock device.
This will allow the VM to be rebooted with a vsock device.
Fixes: #1083
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Generate an error during validation if an attempt it made to place a
device behind an IOMMU or using a VFIO device when not using PCI.
Fixes: #751
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Let's put an underscore "_" in front of each device name to identify
when it has been set internally.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The virtio-console was not added to the list of Migratable devices,
which is fixed from this patch.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This identifier is chosen from the DeviceManager so that it will manage
all identifiers across the VM, which will ensure uniqueness.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This identifier is chosen from the DeviceManager so that it will manage
all identifiers across the VM, which will ensure uniqueness.
It is based off the name from the virtio device attached to this
transport layer.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This identifier is chosen from the DeviceManager so that it will manage
all identifiers across the VM, which will ensure uniqueness.
It is based off the name from the virtio device attached to this
transport layer.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Because we know we will need every virtio device to be identified with a
unique id, we can simplify the code by making the identifier mandatory.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This identifier is chosen from the DeviceManager so that it will manage
all identifiers across the VM, which will ensure uniqueness.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This identifier is chosen from the DeviceManager so that it will manage
all identifiers across the VM, which will ensure uniqueness.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This identifier is chosen from the DeviceManager so that it will manage
all identifiers across the VM, which will ensure uniqueness.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This identifier is chosen from the DeviceManager so that it will manage
all identifiers across the VM, which will ensure uniqueness.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This identifier is chosen from the DeviceManager so that it will manage
all identifiers across the VM, which will ensure uniqueness.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This identifier is chosen from the DeviceManager so that it will manage
all identifiers across the VM, which will ensure uniqueness.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This identifier is chosen from the DeviceManager so that it will manage
all identifiers across the VM, which will ensure uniqueness.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This identifier is chosen from the DeviceManager so that it will manage
all identifiers across the VM, which will ensure uniqueness.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This identifier is chosen from the DeviceManager so that it will manage
all identifiers across the VM, which will ensure uniqueness.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This identifier is chosen from the DeviceManager so that it will manage
all identifiers across the VM, which will ensure uniqueness.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This identifier is chosen from the DeviceManager so that it will manage
all identifiers across the VM, which will ensure uniqueness.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This identifier is chosen from the DeviceManager so that it will manage
all identifiers across the VM, which will ensure uniqueness.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Even in the context of "mmio" feature, we need the next device name to
be generated as we need to identify virtio-mmio devices to support
snapshot and restore functionalities.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This will be later used to identify each device used by the VM in order
to perform introspection and snapshot/restore properly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This will be later used to identify each device used by the VM in order
to perform introspection and snapshot/restore properly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This will be later used to identify each device used by the VM in order
to perform introspection and snapshot/restore properly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This will be later used to identify each device used by the VM in order
to perform introspection and snapshot/restore properly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This will be later used to identify each device used by the VM in order
to perform introspection and snapshot/restore properly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
If the virtio-console device is supposed to be placed behind the virtual
IOMMU, this must be explicitly propagated through the code.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
If the virtio-rng device is supposed to be placed behind the virtual
IOMMU, this must be explicitly propagated through the code.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
If the virtio-vsock device is supposed to be placed behind the virtual
IOMMU, this must be explicitly propagated through the code.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We pass it to the integration and unit tests script through --libc.
Cargo tests are left unmodified.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Now that all virtio devices are assigned with identifiers, they could
all be removed from the VM. This is not something that we want to allow
because it does not make sense for some devices. That's why based on the
device type, we remove the device or we return an error to the user.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The parameters regarding the attachment to the virtio-iommu device was
not propagated correclty, and any modification to the configuration was
not stored back into it.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
It's possible to have multiple vsock devices so in preparation for
hotplug/unplug it is important to be able to have a unique identifier
for each device.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
If the current state is paused that means most of the handles got killed by pthread_kill
We need to unpark those threads to make the shutdown worked. Otherwise
The shutdown API hangs and the API is not responding afterwards. So
before the shutdown call we need to resume the VM make it succeed.
Fixes: #817
Signed-off-by: Muminul Islam <muislam@microsoft.com>
If a size is specified use it (in particular this is required if the
destination is a directory) otherwise seek in the file to get the size
of the file.
Add a new check that the size is a multiple of 2MiB otherwise the kernel
will reject it.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Check that if any device using vhost-user (net & disk with
vhost_user=true) or virtio-fs is enabled then check shared memory is
also enabled.
Fixes: #848
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The new 'shared' and 'hugepages' controls aim to replace the 'file'
option in MemoryConfig. This patch also updated all related integration
tests to use the new controls (instead of providing explicit paths to
"/dev/shm" or "/dev/hugepages").
Fixes: #1011
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Signed-off-by: Bo Chen <chen.bo@intel.com>
FS_IO is part of the actions a vhost-user-fs daemon can ask the VMM to
perform on its behalf. It is meant to read/write the content from a file
descriptor directly into a guest memory region. This region can either
be a RAM region or the dedicated cache region for virtio-fs.
The way FS_IO was implemented, it was only expecting the guest physical
address provided through the "cache_offset" field to refer to the cache
region. Unfortunately, this was only implementing FS_IO partially.
This patch extends the existing FS_IO implementation by checking the GPA
against the cache region as a first step, but if it is not part of the
cache region address range, then we fallback onto searching for a RAM
region that could match. If there is a matching RAM region, we retrieve
the corresponding host address to let the VMM read/write from/to it.
Fixes: #1054
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
The newly added integration test "test_snapshot_restore" is very
unstable, which causes our CI to fail on most pull requests, which is
not acceptable. That's why we ignore this test until we can fix the
stability issue.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that we have multiple virtio devices supporting snapshot and restore
operations, we can add a new integration test to validate the migration
feature works as expected.
The important part is virtio-net as it is used to ssh into the VM to
verify the VM has been restored in the proper state.
This test only works for virtio-mmio for now. Further support for
testing virtio-pci will be added once the virtio-pci transport layer
will support migration.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This doc describe how to use vhost-user-blk in Cloud Hypervisor
with SPDK. With this doc, we can setup simple test environment on
Ubuntu release.
Signed-off-by: Yang Zhong <yang.zhong@intel.com>
This patch implements the Snapshottable trait for virtio-console, which
enables migration support for it. A VM with a virtio-console device
attached can be snapshot and then restored without issues.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Replace alignment calculation of start address with functionally
equivalent version that does not assume that the block size is a power
of two.
Signed-off-by: Martin Xu <martin.xu@intel.com>
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The frame buffer must be updated depending on the amount read from it,
which depends on the number and depth of descriptors available at the
time of the processing.
This patch handles this buffer update, and allow for large buffers to be
correctly processed in multiple rounds.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
On the restore path, using the available and used indexes read from
memory to fill the Queue structure was a mistake. Indeed, the available
index is written from the guest and it reflects the last available index
in the descriptor table. But the driver might have queued a lot of
buffers which have not yet been used by the device. This leads to a
situation where the next_avail from Queue is completely different from
the one we can read from memory.
Instead, the right way to determine the next_avail index that should be
used by the device is by relying on the used index from the memory. This
index represents the correct information we're looking for as it has
been updated before the snapshot to let the guest know the next index to
process.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
First, this modifies the existing helpers on how to get indexes for
available and used rings from memory. Instead of updating the queue
through each helper, they are now used as simple getters.
Based on these new getters, we could create a new helper to determine if
the queue has some available descriptors already queued from the driver
side. This helper is going to be particularly helpful when trying to
determine from a virtio thread if a queue is already loaded with some
available buffers that can be used to send information to the guest.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
When a virtio device is dynamically removed from the VM through the
hot-unplug mechanism, every mapping associated with it must be properly
removed.
Based on the previous patches letting a VirtioDevice expose the list of
userspace mappings associated with it, this patch can now remove all the
KVM userspace memory regions through the MemoryManager.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The same way we added a helper for creating userspace memory mappings
from the MemoryManager, this patch adds a new helper to remove some
previously added mappings.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Since the virtio-fs device is backed by a vhost-user process, it is
important to implement the proper shutdown() function from the
VirtioDevice trait, as vhost-user-blk and vhost-user-net do.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
When hot-unplugging the virtio-pmem from the VM, we don't remove the
associated userspace mapping. This patch will let us fix this in a
following patch. For now, it simply adapts the code so that the Pmem
device knows about the mapping associated with it. By knowing about it,
it can expose it to the caller through the new userspace_mappings()
function.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This will help when we will implement the hot-unplug of the virtio-fs
device, as we will have to remove correctly the userspace mappings
associated with the device.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Introduce new getter function to the VirtioDevice trait, as it will
allow the caller to retrieve the list of userspace mappings associated
with the device.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
There is some duplication between regular and hotplug virtio-fs tests
that can be factorized by adding a simple hotplug flag to choose if each
test should run with or without hotplugging the device.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The hotplugged virtio devices were not added to the list of virtio
devices from the DeviceManager. This patch fixes it, as it was causing
hotplugged virtio-fs devices from not supporting memory hotplug, since
they were never getting the update as they were not part of the list of
virtio devices held by the DeviceManager.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Adds DeviceManager method `make_virtio_fs_device` which creates a single
device, and modifies `make_virtio_fs_devices` to use this method.
Implements the new `vm.add-fs route`.
Signed-off-by: Dean Sheather <dean@coder.com>
Currently unimplemented. Once implemented, this API will allow for
creating virtio-fs devices in the VM after it has booted.
Signed-off-by: Dean Sheather <dean@coder.com>
In the context of the shared memory region used by virtio-fs in order to
support DAX feature, the shared region is exposed as a dedicated PCI
BAR, and it is backed by a KVM userspace mapping.
Upon BAR remapping, the BAR is moved to a different location in the
guest address space, and the KVM mapping must be updated accordingly.
Additionally, we need the VirtioDevice to report the updated guest
address through the shared memory region returned by get_shm_regions().
That's why a new setter is added to the VirtioDevice trait, so that
after the mapping has been updated for KVM, we can tell the VirtioDevice
the new guest address the shared region is located at.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By adding the shared memory regions to the list of BARs, we make sure
the DeviceManager will register it as a BAR on the PCI bus. Without
this, when PCI BAR reprogramming happens, the PCI bus errors since it
does not know about any BAR at the specified address.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The order the elements are pushed into the list is important to restore
them in the right order. This is particularly important for MmioDevice
(or VirtioPciDevice) and their VirtioDevice counterpart.
A device must be fully ready before its associated transport layer
management can trigger its restoration, which will end up activating the
device in most cases.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Any virtio device relying on the mmio transport layer can be snapshotted
and restored thanks to this new patch. From the MmioDevice perspective,
it is mainly a matter of saving the information about the virtqueues as
the restore path will need them to activate the device (if needed
because it has been activated before being snapshotted).
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In anticipation for adding snapshot/restore support to virtio devices,
this commit introduces two new helpers updating the available and used
indexes of a queue, relying on the guest memory.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit relies on serde to serialize and deserialize the content of
a Queue structure. This will be useful information to store when
implementing snapshot/restore feature for virtio devices.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Add the WRITE_KILL_PRIV write flag, corresponding to
FUSE_WRITE_KILL_PRIV introduced in 7.31, and use to only remove the
setuid and setgid bits (by switching credentials) conditionally.
Signed-off-by: Sergio Lopez <slp@redhat.com>
Add support for MAX_PAGES, corresonding to FUSE_MAX_PAGES introduced
in FUSE 7.28.
This allows us to negotiate with the kernel the maximum number of
pages that we support to transfer in a single request.
Signed-off-by: Sergio Lopez <slp@redhat.com>
Add support for FOPEN_CACHE_DIR, a flag that allows us to tell the
guest that it's safe to cache a directory, introduced in FUSE 7.28.
Signed-off-by: Sergio Lopez <slp@redhat.com>
When reporting the BAR size it is necessary to return a value that is
encoded such that all the bits are set that represent the mask of the
natural alignment of the field.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add support for specifying the PCI revision in the PCI configuration and
populate this with the value of 1 for virtio-pci devices.
The virtio-pci specification is slightly ambiguous only saying that
transitional (i.e. devices that support legacy and virtio 1.0) should
set this to 0. In practice it seems that software expects the revision
to be set to 1 for modern only devices.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
OVMF and other standard firmwares use I/O port 0x402 as a simple debug
port by writing ASCII characters to it. This is gated under a feature
that is not enabled by default.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
After all the previous refactoring patches, we can finally create
multiple threads under the same backend. This is directly combined with
multiqueues so that it will create one thread per queue.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Anticipating the follow up patches to run multiple threads for the same
backend, we need the initialization of the disk to happen in the high
level structure VhostUserBlkBackend.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The DiskFile will need to be shared across multiple threads when running
multiple queues across these threads. That's why it needs to be put
inside an Arc. The reason for the Mutex is because execute() expects a
mutable object implementing Read + Write + Seek. Unfortunately, this
create a contention point as the object needs to be locked from each
thread, reducing the performance gain we will get with multiple threads.
The need for an immutable object would solve this problem, and it will
be addressed later through follow up patches.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
There is no need for retrieving the VringWorker since we don't need to
register some extra file descriptors to the epoll loop.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We now support assigning device ids for VFIO and virtio-pci devices so
this error can be generalised.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Extend the eject_device() method on DeviceManager to also support
virtio-pci devices being unplugged.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add an accessor to return the underlying VirtioDevice. This is useful
for managing the removal of the device from internal datastructures when
handling virtio-pci device unplug.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In order to support hotplugging there is a map of human readable device
id to PCI BDF map.
As the device id is part of the specific device configuration (e.g.
NetConfig) it is necessary to return the id through from the helper
functions that create the devices through to the functions that add
those devices to the bus. This necessitates changing a great deal of
function prototypes but otherwise has little impact.
Currently only if an id is supplied by the user as part of the device
configuration is it populated into this map. A later commit will
populate with an autogenerated name where none is supplied by the user.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In order to support freeing the memory that is allocated we need to make
sure that we update the internal representation so that free_bars() can
correctly free the memory if the device has its BARs moved.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Implement the free_bars() method from the PciDevice trait which is used
as part of the device removal process. Although there is only one BAR
allocated by VirtioPciDevice simplify the code by using a vector.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Now that ownership of the memory regions used for the virtio-pmem and
vhost-user-fs devices have been moved into those devices it is no longer
necessary to track them inside DeviceManager.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Move the release of the managed memory region from the DeviceManager to
the vhost-user-fs device. This ensures that the memory will be freed when
the device is unplugged which will lead to it being Drop()ed.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Move the release of the managed memory region from the DeviceManager to
the virtio-pmem device. This ensures that the memory will be freed when
the device is unplugged which will lead to it being Drop()ed.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
By implementing queues_per_thread(), this patch fills the last missing
bit to enable multithreaded multiqueue support for the vhost-user-net
backend implementation.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By adding a "thread_id" parameter to handle_event(), the backend crate
can now indicate to the backend implementation which thread triggered
the processing of some events.
This is applied to vhost-user-net backend and allows for simplifying a
lot the code since each thread is identical.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By adding the "thread_index" parameter to the function exit_event() from
the VhostUserBackend trait, the backend crate now has the ability to ask
the backend implementation about the exit event related to a specific
thread.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to prepare for the support of multithreaded multiqueues, the
structure VhostUserNetThread is simplified to hold only one RX queue,
one TX queue, and one TAP interface.
Following this change, VhostUserNetBackend now holds a list of threads
instead of going through each thread to handle multiqueues.
These changes decouple neatly the abstraction between the backend and
each thread. This allows for a lot of simplification since we now know
all threads are identical, hence the handling of events becomes very
straightforward.
One important point is that each thread can be locked when in use,
without causing any contention with other threads since the backend
doesn't need to be locked anymore.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that multiple worker threads can be run from the backend crate, it
is important that each backend implementation can access every worker
thread.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to support multiqueues running on multiple threads for
increasing the IO performances, this commit introduces a new function
queues_per_thread() to the VhostUserBackend trait.
This gives each backend implementation the opportunity to define which
queues belong to which thread.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By changing the mutability of this function, after adapting all
backends, we should be able to implement multithreads with
multiqueues support without hitting a bottleneck on the backend
locking.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Both blk, net and fs backends have been updated to avoid the requirement
of having handle_event(&mut self). This will allow the backend crate to
avoid taking a write lock onto the backend object, which will remove the
potential contention point when multiple threads will be handling
multiqueues.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
While testing self spawned vhost-user backends, it appeared that the
backend was aborting due to a missing system call in the seccomp
filters. mremap() was the culprit and this patch simply adds it to the
whitelist.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This should make it a pointer in the Go generated code so that it will
be ommitted and thus not populated with an unhelpful default value.
Fixes: #1015
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This ensures that the field is filled with None when it is not specified
as part of the deserialisation step.
Fixes: #1015
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
We can now allow guests that specify an initramfs to boot
using the PVH boot protocol.
Signed-off-by: Alejandro Jimenez <alejandro.j.jimenez@oracle.com>
Fill and write to guest memory the necessary boot module
structure to allow a guest using the PVH boot protocol
to load an initramfs image.
Signed-off-by: Alejandro Jimenez <alejandro.j.jimenez@oracle.com>
This allows the validation of this requirement for both command line
booted VMs and those booted via the API.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This allows the validation of this requirement for both command line
booted VMs and those booted via the API.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
As the VmConfig::Parse() also does validation work it only make sense to
parse the VM options on the VM boot path only.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Replace the existing VmConfig::valid() check with a call into
.validate() as part of earlier config setup or boot API checks.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When performing an API boot validate the configuration. For now only
some very basic validation is performed but in subsequent commits
the validation will be extended.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The configuration comes from a variety of places (commandline, REST API
and restore) however some validation was only happening on the command
line parsing path. Therefore introduce a new ability to validate the
configuration before proceeding so that this can be used for commandline
and API boots.
For now move just the console and serial output mode validation under
the new validation API.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Making sure the OpenAPI definition is up to date with newly added
structure and parameters to support VM restoration.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that the restore path uses RestoreConfig structure, we add a new
parameter called "prefault" to it. This will give the user the ability
to populate the pages corresponding to the mapped regions backed by the
snapshotted memory files.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The goal here is to move the restore parameters into a dedicated
structure that can be reused from the entire codebase, making the
addition or removal of a parameter easier.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
When CoW can be used, the VM restoration time is reduced, but the pages
are not populated. This can lead to some slowness from the guest when
accessing these pages.
Depending on the use case, we might prefer a slower boot time for better
performances from guest runtime. The way to achieve this is to prefault
the pages in this case, using the MAP_POPULATE flag along with CoW.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This patch extends the previous behavior on the restore codepath.
Instead of copying the memory regions content from the snapshot files
into the new memory regions, the VMM will use the snapshot region files
as the backing files behind each mapped region. This is done in order to
reduce the time for the VM to be restored.
When the source VM has been initially started with a backing file, this
means it has been mapped with the MAP_SHARED flag. For this case, we
cannot use the CoW trick to speed up the VM restore path and we simply
fallback onto the copy of the memory regions content.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Whenever a MemoryManager is restored from a snapshot, the memory regions
associated with it might need to directly back the mapped memory for
increased performances. If that's the case, a list of external regions
is provided and the MemoryManager should simply ignore what's coming
from the MemoryConfig.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that we can choose specific mmap flags for the guest RAM, we create
a new parameter "copy_on_write" meaning that the memory mappings backed
by a file should be performed with MAP_PRIVATE instead of MAP_SHARED.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to anticipate the need for special mmap flags when memory
mapping the guest RAM, we need to switch from from_file() wrapper to
build() wrapper.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
A few KVM ioctls were missing in order to perform both snapshot and
restore while keeping seccomp enabled.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This connects the dots together, making the request from the user reach
the actual implementation for restoring the VM.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This connects the dots together, making the request from the user reach
the actual implementation for snapshotting the VM.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The MemoryManager is somehow a special case, as its restore() function
was not implemented as part of the Snapshottable trait. Instead, and
because restoring memory regions rely both on vm.json and every memory
region snapshot file, the memory manager is restored at creation time.
This makes the restore path slightly different from CpuManager, Vcpu,
DeviceManager and Vm, but achieve the correct restoration of the
MemoryManager along with its memory regions filled with the correct
content.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
This is only implementing the send() function in order to store all Vm
states into a file.
This needs to be extended for live migration, by adding more transport
methods, and also the recv() function must be implemented.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
By aggregating snapshots from the CpuManager, the MemoryManager and the
DeviceManager, Vm implements the snapshot() function from the
Snapshottable trait.
And by restoring snapshots from the CpuManager, the MemoryManager and
the DeviceManager, Vm implements the restore() function from the
Snapshottable trait.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
This implements the send() function of the Transportable trait, so that
the guest memory regions can be saved into one file per region.
This will need to be extended for live migration, as it will require
other transport methods and the recv() function will need to be
implemented too.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
In order to snapshot the content of the guest RAM, the MemoryManager
must implement the Snapshottable trait.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Implement the Snapshottable trait for Vcpu, and then implements it for
CpuManager. Note that CpuManager goes through the Snapshottable
implementation of Vcpu for every vCPU in order to implement the
Snapshottable trait for itself.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Anticipating the need for a slightly different function for restoring
vCPUs, this patch factorizes most of the vCPU creation, so that it can
be reused for migration purposes.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
These two new helpers will be useful to capture a vCPU state and being
able to restore it at a later time.
Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
In anticipation for the CpuManager to aggregate all Vcpu snapshots
together, this change makes sure the CpuManager has a handle onto
every vCPU.
Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Based on the list of Migratable devices stored by the DeviceManager, the
DeviceManager can implement the Snapshottable trait by aggregating all
devices snapshots together.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Serial and Ioapic both implement the Migratable trait, hence the
DeviceManager can store them in the list of Migratable devices.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
We need the project to rely on kvm-bindings and kvm-ioctls branches
which include the serde derive to be able to serialize and deserialize
some KVM structures.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The parse_size helper function can now be consolidated into the
ByteSized FromStr implementation.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Now all parsing code makes use of the Toggle and it's FromStr support
move the helper function into the from_str() implementation.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The integration tests and documentation make use of empty value strings
like "--net tap=" accept them but return None so that the default value
will be used as expected.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This simplifies the parsing of the option by using OptionParser along
with its automatic conversion behaviour.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Byte sizes are quantities ending in "K", "M", "G" and by implementing
this type with a FromStr implementation the values can be converted
using .parse().
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Before porting over to OptionParser add a unit test to validate the
current memory parsing code. This showed up a bug where the "size=" was
always required. Temporarily resolve this by assigning the string a
default value which will later be replaced when the code is refactored.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This will be used to simplify and consolidate much of the parsing code
used for command line parameters.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
A Snapshottable component can snapshot itself and
provide a MigrationSnapshot payload as a result.
A MigrationSnapshot payload is a map of component IDs to a list of
migration sections (MigrationSection). As component can be made of
several Migratable sub-components (e.g. the DeviceManager and its
device objects), a migration snapshot can be made of multiple snapshot
itself.
A snapshot is a list of migration sections, each section being a
component state snapshot. Having multiple sections allows for easier and
backward compatible migration payload extensions.
Once created, a migratable component snapshot may be transported and this
is what the Transportable trait defines, through 2 methods: send and recv.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
On some systems, the open() system call is used by Cloud-Hypervisor,
that's why it should be part of the seccomp filters whitelist.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Both clock_gettime and gettimeofday syscalls where missing when running
Cloud-Hypervisor on a Linux host without vDSO enabled. On a system with
vDSO enabled, the syscalls performed by vDSO were not filtered, that's
why we didn't have to whitelist them.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Return libc::EINVAL instead of custom "Wrong offset" error, as mmap(2)
returns EINVAL when offset/len is invalid.
Signed-off-by: Eryu Guan <eguan@linux.alibaba.com>
In fs_slave_map/unmap/sync, we only made sure offset < cache_size, but
didn't validate (offset + len). We should ensure [offset, offset+len]
is within cache range as well.
Signed-off-by: Eryu Guan <eguan@linux.alibaba.com>
One more time, we're extending the VFIO integration test to verify that
a VFIO device passed through a VM is still usable after some new memory
has been hotplugged.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Extend the update_memory() method from DeviceManager so that VFIO PCI
devices can update their DMA mappings to the physical IOMMU, after a
memory hotplug has been performed.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Whenever the memory is resized, it's important to retrieve the new
region to pass it down to the device manager, this way it can decide
what to do with it.
Also, there's no need to use a boolean as we can instead use an Option
to carry the information about the region. In case of virtio-mem, there
will be no region since the whole memory has been reserved up front by
the VMM at boot. This means only the ACPI hotplug will return a region
and is the only method that requires the memory to be updated from the
device manager.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Whenever the guest memory is extended, the IOMMU mappings related to
each VFIO device must be updated. This allows subsequent RAM accesses
to be authorized by the physical IOMMU for the newly added RAM region.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Commit 2adddce2 reorganized the crate for a cleaner multi architecture
(x86_64 and aarch64) support.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
For now, the codebase does not support booting from initramfs with PVH
boot protocol, therefore we need to fallback to the legacy boot.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
the integration test creates an initramfs image based on AlpineLinux mini root filesystem
with a simple /init script that just echoes a string to the console. The string
is passed via the kernel cmdline as an environment variable.
Signed-off-by: Damjan Georgievski <gdamjan@gmail.com>
* load the initramfs File into the guest memory, aligned to page size
* finally setup the initramfs address and its size into the boot params
(in configure_64bit_boot)
Signed-off-by: Damjan Georgievski <gdamjan@gmail.com>
currently unused, the initramfs argument is added to the cli,
and stored in vmm::config:VmConfig as an Option(InitramfsConfig(PathBuf))
Signed-off-by: Damjan Georgievski <gdamjan@gmail.com>
The persistent memory will be hotplugged via DeviceManager and saved in
the config for later use.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Split it into a method that creates a single device which is called by
the multiple device version so this can be used when dynamically adding
a device.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This commit adds new option hotplug_method to memory config.
It can set the hotplug method to "acpi" or "virtio-mem".
Signed-off-by: Hui Zhu <teawater@antfin.com>
The basic idea of virtio-mem is to provide a flexible, cross-architecture
memory hot plug and hot unplug solution that avoids many limitations
imposed by existing technologies, architectures, and interfaces. More
details can be found in https://lkml.org/lkml/2019/12/12/681.
This commit add virtio-mem device.
Signed-off-by: Hui Zhu <teawater@antfin.com>
The persistent memory will be hotplugged via DeviceManager and saved in
the config for later use.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Split it into a method that creates a single device which is called by
the multiple device version so this can be used when dynamically adding
a device.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
We made sure gpa is in cache range, but not the end addr of request,
which is (gpa + len). If the end addr of request is beyond dax cache
window, vmm would corrupt guest memory or crash.
Fix it by making sure end addr of request is within cache range as well.
And while we're at it, return EFAULT if the request is out of range, as
write(2)/read(2) returns EFAULT when buffer is outside accessible
address space.
Signed-off-by: Eryu Guan <eguan@linux.alibaba.com>
Split it into a method that creates a single device which is called by
the multiple device version so this can be used when dynamically adding
a device.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
All vhost-user-blk, vhost-user-net and virtio-fs integration tests are
extended with memory resizing. The point is to validate that we can
hotplug some memory and keep these backends to work as expected.
Just a note that because memory resizing is not supported by mmio, we
limit these extensions to all non-mmio use cases.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The same Clear Linux kernel command line was repeated again and again in
the code. That's why this commit takes care of factorizing it, which
simplifies the code but also simplify maintenance whenever we'd need to
update the block UUID.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The hypervisor-fw does not support virtio-blk through mmio transport
layer, therefore we can only run tests with mmio if these tests boot
directly to kernel.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
There's no reason mmio builds cannot support vhost-user-blk, therefore
there's no reason to skip these tests for mmio.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
As a first step towards factorizing as many vhost-user-blk tests as
possible, this patch takes care of unifying their expected output.
It also makes a clear separation between the tests booting from
vhost-user-blk from the ones passing an extra volume.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to limit code duplication, vhost-user-net integration tests are
further factorized, as they now include the self spawning test.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to keep vhost-user backend to work across guest memory resizing
happening when memory is hot-plugged or hot-unplugged, both blk, net and
fs devices are implementing the notifier to let the backend know.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By factorizing the setup of the memory table for vhost-user, we
anticipate the fact that vhost-user devices are going to reuse this
function when the guest memory will be updated.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Whenever the VM memory is resized, DeviceManager needs to be notified
so that it can subsequently notify each virtio devices about it.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The virtio devices backed by a vhost-user backend must send an update to
the associated backend with the new file descriptors corresponding to
the memory regions.
This patch allows such devices to be notified when such update needs to
happen.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This separates the filters used between the VMM and API threads, so that
we can apply different rules for each thread.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This change introduces a new CLI option --seccomp. This allows the user
to enable/disable the seccomp filters when needed. Because the user now
has the possibility to disable the seccomp filters, and because the
Cloud-Hypervisor project wants to enforce the maximum security by
default, the seccomp filters are now applied by default.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit introduces the application of the seccomp filter to the VMM
thread. The filter is empty for now (SeccompLevel::None).
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the seccomp crate, we create a new vmm module responsible for
creating a seccomp filter that will be applied to the VMM main thread.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The seccomp crate from Firecracker is nicely implemented, documented and
tested, which is a good reason for relying on it to create and apply
seccomp filters.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Virtio-fs daemon expects fs_slave_io() returns the number of bytes
read/written on success, but we always return 0 and make userspace think
nothing has been read/written.
Fix it by returning the actual bytes read/written. Note that This
depends on the corresponding fix in vhost crate.
Fixes: #949
Signed-off-by: Eryu Guan <eguan@linux.alibaba.com>
Factorise test_virtio_pmem to test changes are not persisted if file is
readonly.
As virtio-mem is now supported in the upstream kernel we can switch to
using the firmware.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Rather than using a raw OS disk image. This will be useful when the test
is extended to doing I/O on the image.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This opens the backing file read-only, makes the pages in the mmap()
read-only and also makes the KVM mapping read-only. The file is also
mapped with MAP_PRIVATE to make the changes local to this process only.
This is functional alternative to having support for making a
virtio-pmem device readonly. Unfortunately there is no concept of
readonly virtio-pmem (or any type of NVDIMM/PMEM) in the Linux kernel so
to be able to have a block device that is appears readonly in the guest
requires significant specification and kernel changes.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Use this boolean to turn on the KVM_MEM_READONLY flag to indicate that
this memory mapping should not be writable by the VM.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
According to `asm-generic/termios.h`, the `struct winsize` should be:
struct winsize {
unsigned short ws_row;
unsigned short ws_col;
unsigned short ws_xpixel;
unsigned short ws_ypixel;
};
The ioctl of TIOCGWINSZ will trigger a segfault on aarch64.
Signed-off-by: Qiu Wenbo <qiuwenbo@phytium.com.cn>
This feature is stable and there is no need for this to be behind a
flag. This will also reduce the time needed to run the integration test
as we will not be running them all again under the flag.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This table currently contains only all the VFIO devices and it should
really contain all the PCI devices.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Previously this was only returned if the device had an IOMMU mapping and
whether the device should be added to the virtio-iommu. This was already
captured earlier as part of creating the device so use that information
instead.
Always returning the B/D/F is helpful as it facilitates virtio PCI
device hotplug.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Implement missing support for FUSE_LSEEK, which basically implies
calling to libc::lseek on the file handle. As this operation alters
the file offset, we take a write lock on the File's RwLock.
Signed-off-by: Sergio Lopez <slp@redhat.com>
Add a document to illustrate how to test vhost-user-net function with
OpenVSwitch + DPDK as backend.
Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
On x86_64, a hint to the compiler is not enough, we need to issue a
MFENCE instruction. Replace the Acquire fence with a SeqCst one.
Without this, it's still possible to miss an used_event update,
leading to the omission of a notification, possibly stalling the
vring.
Signed-off-by: Sergio Lopez <slp@redhat.com>
I spent a few minutes trying to understand why we were unconditionally
updating the VM config memory size, even if the guest memory resizing
did not happen.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
We set it to 0xff, which is for unregistered loaders.
The kernel checks that the bootloader ID is set when e.g. loading
ramdisks, so not setting it when we get a bootparams header from the
loader will prevent the kernel from loading ramdisks.
Fixes: #918
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The IORT table for virtio-iommu use was removed and replaced with a
purely virtio based solution. Although the table construction was
removed these structures were left behind.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Virtiofs's dax window can be used as read/write's source (e.g. mmap a file
on virtiofs), but the dax window area is not shared with vhost-user
backend, i.e. virtiofs daemon.
To make those IO work, addresses of this kind of IO source are routed to
VMM via FS_IO requests to perform a read/write from an fd directly to the
given GPA.
This adds the support of FS_IO request to clh's vhost-user-fs master part.
Signed-off-by: Liu Bo <bo.liu@linux.alibaba.com>
Add an integration test that builds cloud-hypervisor with
the pvh_boot feature and boots a kernel built with CONFIG_PVH.
Signed-off-by: Alejandro Jimenez <alejandro.j.jimenez@oracle.com>
Validate correct GDT entries, initial segment configuration, and control
register bits that are required by PVH boot protocol.
Signed-off-by: Alejandro Jimenez <alejandro.j.jimenez@oracle.com>
Expand the unit tests to cover the configure_system() code when
using the PVH boot protocol. Verify the method for adding memory
map table entries in the format specified by PVH boot protocol.
Signed-off-by: Alejandro Jimenez <alejandro.j.jimenez@oracle.com>
Use a new feature called "pvh_boot" to enable using the PVH boot
protocol if the guest kernel supports it. The feature can be enabled
by building with:
cargo build [--release] --features "pvh_boot"
Once performance has been evaluated, this can be made part of the
default set of features so that any guest that supports it boots
using PVH as the preferred option as is the case in QEMU.
Signed-off-by: Alejandro Jimenez <alejandro.j.jimenez@oracle.com>
Fill the hvm_start_info and related memory map structures as
specified in the PVH boot protocol. Write the data structures
to guest memory at the GPA that will be stored in %rbx when
the guest starts.
Signed-off-by: Alejandro Jimenez <alejandro.j.jimenez@oracle.com>
In order to properly initialize the kvm regs/sregs structs for
the guest, the load_kernel() return type must specify which
boot protocol to use with the entry point address it returns.
Make load_kernel() return an EntryPoint struct containing the
required information. This structure will later be used
in the vCPU configuration methods to setup the appropriate
initial conditions for the guest.
Signed-off-by: Alejandro Jimenez <alejandro.j.jimenez@oracle.com>
Create supporting definitions to use the hvm start info and memory
map table entry struct definitions from the linux-loader crate in
order to enable PVH boot protocol support
Signed-off-by: Alejandro Jimenez <alejandro.j.jimenez@oracle.com>
@dependabot bumped the dependency to 0.4.10 but this is no longer a
valid version so downgrade appropriately.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Extended attributes (xattr) support has a huge impact on write
performance. The reason for this is that, if enabled, FUSE sends a
setxattr request after each write operation, and due to the inode
locking inside the kernel during said request, the ability to execute
the operations in parallel becomes heavily limited.
Signed-off-by: Sergio Lopez <slp@redhat.com>
This change enables vhost_user_fs to process multiple requests in
parallel by scheduling them into a ThreadPool (from the Futures
crate).
Parallelism on a single file is limited by the nature of the operation
executed on it. A recent commit replaced the Mutex that protects the
File within HandleData with a RwLock, to allow some operations (at
this moment, only "read" and "write") to proceed in parallel by
acquiring a read lock.
A more complex approach was also implemented [1], involving
instrumentation through vhost_user_backend to be able to serialize
completions, reducing the pressure on the vring RwLock. This strategy
improved the performance on some corner cases, while making it worse
on other, more common ones. This fact, in addition to it requiring
wider changes through the source code, prompted me to drop it in favor
of this one.
[1] https://github.com/slp/cloud-hypervisor/tree/vuf_async
Signed-off-by: Sergio Lopez <slp@redhat.com>
"DescriptorChain"s are tied to the lifetime of the referenced
GuestMemoryMmap object (for good reasons), but sometimes (i.e., when
processing descriptors from different contexts) we may need to switch
them to point a different GuestMemoryMmap.
Here we introduce the structure DescriptorHead, which holds the data
needed to rebuild a DescriptorChain, the method "get_head" which
returns the DescriptorHead for a DescriptorChain, and the method
"new_from_head", which allows to create a new DescriptorChain with a
DescriptorHead and a new reference to a GuestMemoryMmap.
Signed-off-by: Sergio Lopez <slp@redhat.com>
Replace HandleData's File Mutex with a RwLock to have more granularity
on the lock. This allows operations on the same File that are safe to
be run in parallel (at this moment, read and write), to acquire a read
lock to avoid waiting on each other.
Signed-off-by: Sergio Lopez <slp@redhat.com>
When using "--disk" with a vhost socket and not using self spawning then
it is not necessary or helpful to specify the path.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This option was superseded by using "--net" with "vhost_user=true". This
option wasn't being parsed any more but was left over.
Fixes: #806
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
By using a Vec to hold the list of devices on the PciBus, there's a
problem when we use unplug. Indeed, the vector of devices gets reduced
and if the unplugged device was not the last one from the list, every
other device after this one is shifted on the bus.
To solve this problem, a HashMap is used. This allows to keep track of
the exact place where each device stands on the bus.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The option desired_ram is in byte, make larger the amount of memory to
add.
Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
Some of the help strings had extra newlines in them or otherwise strange
wrapping. The strings were rewrapped with the nightly version of rustfmt
that supports string formatting.
Fixes: #899
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This prevents the output being wrapped at 120 characters and giving
strange results.
Fixes: #899
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This change, combined with the compiler hint to inline get_used_event,
shortens the window between the memory read and the actual check by
calling get_used_event from needs_notification.
Without it, when putting enough pressure on the vring, it's possible
that a notification is wrongly omitted, causing the queue to stall.
Signed-off-by: Sergio Lopez <slp@redhat.com>
get_used_event is used from vhost_user_backend:needs_notification to
check whether an interrupt must be sent to the guest to notify there
are new items in the queue. Shorten the update window by asking the
the compiler to inline this method, so a write won't slip between the
read of the memory contents and the actual check.
Signed-off-by: Sergio Lopez <slp@redhat.com>
Since we only keep one single version of the kernel config file in our
repository, there is no reason to keep the filename complex.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The kernel version is updated from 5.5-rc1 to 5.6-rc4, including the
updated kernel config file.
The kernel branch contains virtio-fs, virtio-iommu and virtio-mem
patches that are not upstream yet. It also contains one fix for
virtio-vsock which will be merged upstream in the next release.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We define 1 workflow with 2 jobs: One for the regular build and another
one for creating a release and uploading the corresponding assets.
They both run conditionally, depending on the gihtub event that triggered
the action.
Fixes: #825
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
With some of the factorization that happened to be able to support VFIO
hotplug, one mistake was made. In case a vIOMMU is created through a
virtio-iommu device, and no matter the "iommu" option value from the
VFIO device parameter, the VFIO device was always placed behind the
virtual IOMMU.
This commit fixes this wrong behavior by making sure the device
configuration is taken into account to decide if it should be attached
or not to the virtual IOMMU.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Add a new id option to the VFIO hotplug command so that it matches the
VFIO coldplug semantic.
This is done by refactoring the existing code for VFIO hotplug, where
VmAddDeviceData structure is replaced by DeviceConfig. This structure is
the one used whenever a VFIO device is coldplugged, which is why it
makes sense to reuse it for the hotplug codepath.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Add the ability to specify the "id" associated with a device, by adding
an extra option to the parameter --device.
This new option is not mandatory, and by default, the VMM will take care
of finding a unique identifier.
If the identifier provided by the user through this new option is not
unique, an error will be thrown and the VM won't be started.
Fixes#881
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The 32 bits MMIO address space is handled separately from the 64 bits
one. For this reason, we need to invoke the appropriate freeing function
to remove a range from this address space.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that PciDevice trait has a dedicated function to remove the bars,
the DeviceManager can invoke this function whenever a PCI device is
unplugged from the VM.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Whenever a VfioPciDevice is dropped, the regions previously mapped are
being unmapped. But there's also a need for removing the region from a
KVM perspective.
This commit extends the existing unmap_mmio_regions() function to take
care of removing the KVM region at the same time the regions are being
unmapped.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to provide the tools for a complete cleanup whenever a VFIO PCI
device is removed from the VM, the VfioPciDevice implements free_bars()
method from PciDevice trait. This will take care of removing the IO and
MMIO ranges previously reserved through the vm-allocator.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The allocator already had functions to allocate and free both IO and 64
bits MMIO address spaces, but it only had an allocating function for 32
bits MMIO address space.
With this new function, it will be now possible to remove cleanly some
ranges from the 32 bits MMIO address space.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The point of this new method is to let the caller decide when the
implementation of the PciDevice should free the BARs previously
allocated through the other method allocate_bars().
This provides a way to perform proper cleanup for any PCI device.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Support sending a request body this will usually be JSON encoded data
representing the details of the request.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Upon removal of a PCI device, make sure we don't hold onto the device ID
as it could be reused for another device later.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to handle the case where devices are very often plugged and
unplugged from a VM, we need to handle the PCI device ID allocation
better.
Any PCI device could be removed, which means we cannot simply rely on
the vector size to give the next available PCI device ID.
That's why this patch stores in memory the information about the 32
slots availability. Based on this information, whenever a new slot is
needed, the code can correctly provide an available ID, or simply return
an error because all slots are taken.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit ensures that when a VFIO device is hot-unplugged from the
VM, it is also removed from the VmConfig. This prevents a potential
reboot from creating the device.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Add a new field to the DeviceConfig, allowing the VMM to allocate a name
to the VFIO devices.
By identifying a VFIO device with a unique name, we can make sure a user
can properly unplug it at any time.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit introduces the new command "remove-device" that will let a
user hot-unplug a VFIO PCI device from an already running VM.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit implements the eject function so that a VFIO device will be
removed from any bus it might sit on, and from any list it might be
stored in.
The idea is to reach a point where there is no reference of the device
anywhere in the code, so that the Drop implementation will be invoked
and so that the device will be fully removed from the VMM.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
When the guest OS is done removing a PCI device, it will invoke the _EJ0
method from ACPI, associated with the device. This will trigger a port
IO write to a region known by the VMM. Upon this writing, the VMM will
trap the VM exit and retrieve the written value.
Based on the value, the VMM will invoke its eject_device() method to
finalize the removal of the device.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The same BusDevice can be inserted multiple times on the same bus at
different ranges. This is the case for a PCI device with multiple memory
BARs for example, as each BAR will be inserted at a specific MMIO range.
This new method allows to find and remove every occurence of the same
device so that we can make sure it is fully removed from the bus.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Simple function relying on the retain() method from std::Vec, allowing
to remove every occurence of the same device.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
As we try to keep track of every PCI device related to the VM, we don't
want to have separate lists depending on the concrete type associated
with the PciDevice trait. Also, we want to be able to cast the actual
type into any trait or concrete type.
The most efficient way to solve all these issues is to store every
device as an Arc<dyn Any + Send + Sync>. This gives the ability to
downcast into the appropriate concrete type, and then to cast back into
any trait that we might need.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Add a new list storing the device names across the entire codebase. VFIO
devices are added to the list whenever a new one is created. By default,
each VFIO device is given a name "vfioX" where X is the first available
integer.
Along with this new list of names, another list is created, grouping PCI
device's name with its associated b/d/f. This will be useful to keep
track of the created devices so that we can implement unplug
functionality.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
When a virtio device is paused an event is written to the appropriate
"pause" EventFd for the device. This will be noticed by the the device's
epoll_wait(), an atomic bool checked an if true then the thread is
parked(). When resuming the bool is reset and the thread is unpark()ed.
However the event triggering the pause is still in the EventFd so the
epoll_wait() will continue to return but because the boolean is not set
the thread will not be park()ed but instead we will busy loop around an
event that is not being consumed.
The solution is to drain the "pause" EventFd when the event is first
received and thus the epoll_wait() will only return for the pause event
once. This resolves the infinite epoll_wait() wake-ups.
Fixes: #869
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This commit introduces a basic implementation of a remote control of a
running VMM implementing a subset of the API.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
If a new build request for master comes in don't cancel the older
builds. Cancelling older builds is very useful for PRs that are being
updated but less so for a build of the master branch that has been
triggered via a merge or a periodic build.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
For cases where IOMMU is not supported, iommu_group will be no present.
This could happened because IOMMU is off at kernel level, bios level or
is not supported at all.
Instead of fail with a generic error handle it as different use case:
Before:
ReadLink(Os { code: 2, kind: NotFound, message: "No such file or directory"})
After:
VfioNoIommuGroup("/sys/bus/pci/devices/0000:03:00.0/iommu_group")
Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
Replace sudo invocations of bash creating a subshell with simpler
solutions by running the program directly on the input or using tee.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
As part of bringing the tests up with virtio-mmio the virtio-fs were
wrongly disabled. Re-enable them and remove the cache size check as the
cache does not appear in /proc/iomem on virtio-mmio.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When booting from vhost-user-block the entropy is sometimes lower
triggering a flaky test. We have already use other, more reliable
methods for checking that the VM has booted.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The Vm structure was used to store a strong reference to the IO bus.
This is not needed anymore since the AddressManager is logically the
one holding this strong reference. This has been made possible by the
introduction of Weak references on the Bus structure itself.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that the BusDevice devices are stored as Weak references by the
IO and MMIO buses, there's no need to use Weak references from the
DeviceManager anymore.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that the BusDevice devices are stored as Weak references by the
IO and MMIO buses, there's no need to use Weak references from the
CpuManager anymore.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that the BusDevice devices are stored as Weak references by the IO
and MMIO buses, there's no need to use Weak references from the PciBus
anymore.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By storing a list of Weak references to BusDevice devices, the Bus
structure prevents potential cyclic dependencies from happening on
strong references.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The point is to make sure the DeviceManager holds a strong reference of
each BusDevice inserted on the IO and MMIO buses. This will allow these
buses to hold Weak references onto the BusDevice devices.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The method add_vfio_device() from the DeviceManager needs to be mutable
if we want later to be able to update some internal fields from the
DeviceManager from this same function.
This commit simply takes care of making the necessary changes to change
this function as mutable.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
It's more logical to name the field referring to the DeviceManager as
"device_manager" instead of "devices".
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit extends the existing test_vfio by hotplugging an extra
virtio-net device to the L2 VM. The test for validating the hotplug
succeeded is the same as the one to verify the non-hotplugged devices.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By inserting the DeviceManager on the IO bus, we introduced some cyclic
dependency:
DeviceManager ---> AddressManager ---> Bus ---> BusDevice
^ |
| |
+---------------------------------------------+
This cycle needs to be broken by inserting a Weak reference instead of
an Arc (considered as a strong reference).
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Ensures the configuration is updated after a new device has been
hotplugged. In the event of a reboot, this means the new VM will be
started with the new device that had been previously hotplugged.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit finalizes the VFIO PCI hotplug support, based on all the
previous commits preparing for it.
One thing to notice, this does not support vIOMMU yet. This means we can
hotplug VFIO PCI devices, but we cannot attach them to an existing or a
new virtio-iommu device.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This factorization is very important as it will allow both the standard
codepath and the VFIO PCI hotplug codepath to rely on the same function
to perform the addition of a new VFIO PCI device.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Whenever the user wants to hotplug a new VFIO PCI device, the VMM will
have to trigger a hotplug notification through the GED device.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit introduces the new command "add-device" that will let a user
hotplug a VFIO PCI device to an already running VM.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Through the BusDevice implementation from the DeviceManager, and by
inserting the DeviceManager on the IO bus for a specific IO port range,
the VMM now has the ability to handle PCI device hotplug.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In anticipation of inserting the DeviceManager on the IO/MMIO buses,
the DeviceManager must implement the BusDevice trait.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Expands the existing GED device description in the DSDT table. A new
case of hotplug notification is being added to the already existing CPU
and memory hotplug. The third case of hotplug being the addition/removal
of a PCI device, it is identified through the flag 0x4.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Create a small method that will perform both hotplug of all the devices
identified by PCIU bitmap, and then perform the hotunplug of all the
devices identified by the PCID bitmap.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The _EJ0 method provides the guest OS a way to notify the VMM that the
device has been properly ejected from the guest OS. Only after this
point, the VMM can fully remove the device.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This new PHPR device in the DSDT table introduces some specific
operation regions and the associated fields.
PCIU stands for "PCI up", which identifies PCI devices that must be
added.
PCID stands for "PCI down", which identifies PCI devices that must be
removed.
B0EJ stands for "Bus 0 eject", which identifies which device on the bus
has been ejected by the guest OS.
Thanks to these fields, the VMM and the guest OS can communicate while
performing hotplug/hotunplug operations.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Adds the DVNT method to the PCI0 device in the DSDT table. This new
method is responsible for checking each slot and notify the guest OS if
one of the slots is supposed to be added or removed.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit introduces the ACPI support for describing the 32 device
slots attached to the main PCI host bridge.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that virtio-mmio transport has support for SHM regions we should be
able to test virtio-fs against it.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The details of the SHM regions or the lack of, which is used by
virtio-fs DAX, is communicated through configuration fields on the
virtio-mmio memory region. Implement the necessary fields to return the
SHM entries and in particular return a length of (u64)-1 which is used
by the kernel to indicate there are no SHM regions.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
A new ClearLinux image has been uploaded to the Azure storage account.
It is based off of the ClearLinux cloudguest image 31310 version, with
three extra bundles added to it.
First bundle is curl, which adds the curl binary to the image, second
bundle is iperf, adding the iperf binary to the image, and third bundle
is sysadmin-basic to include utility like netcat.
The image is 2G in size.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
As we switched to external vhost_rs crate, delete this internal one.
All vhost changes should go to cloud-hypervisor/vhost.
Signed-off-by: Eryu Guan <eguan@linux.alibaba.com>
As cloud-hypervisor/vhost crate (dragonball branch) is ready to be used,
switch vhost_rs from internal crate to the external one.
Signed-off-by: Eryu Guan <eguan@linux.alibaba.com>
Current device configuration space offset value is 0, we need to
update that value to VHOST_USER_CONFIG_OFFSET(0x100) to follow the spec
Fixes#844
Signed-off-by: Arron Wang <arron.wang@intel.com>
On tests that expect a clean shutdown there is no need to try and kill
the child after wait() has returned as the process has already exited.
Further there is no need to sleep before wait() as wait will block until
the VM and VMM shutdown is complete.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This should address any flakiness as the VMM process will have
completely terminated and all files closed.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Many of the tests attempted to SSH into the VM and then run "shutdown"
but don't actually check that the VM has shutdown correctly and proceed
to kill the child process. Remove the associated SSH commands and sleeps
from those tests that are not explicitly checking the shutdown
behaviour.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Only some tests require the output for the tests to be captured so
default to not capturing the output to a pipe and instead make it
controllable.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Many of the tests use an identical network configuration. Add a
GuestCommand::default_net() to generate this configuration and use it
wherever possible.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Many of the tests use an identical disk configuration. Add a
GuestCommand::default_disks() to generate this configuration and use it
wherever possible.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This is a thin wrapper over std::process:Command which currently only
specifies the default binary but in future will handle more default
behaviour.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In anticipation of the support for device hotplug, this commit moves the
DeviceManager object into an Arc<Mutex<>> when the DeviceManager is
being created. The reason is, we need the DeviceManager to implement the
BusDevice trait and then provide it to the IO bus, so that IO accesses
related to device hotplug can be handled correctly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We weren't processing events arriving at the HIPRIO queue, which
implied ignoring FUSE_INTERRUPT, FUSE_FORGET, and FUSE_BATCH_FORGET
requests.
One effect of this issue was that file descriptors weren't closed on
the server, so it eventually hits RLIMIT_NOFILE. Additionally, the
guest OS may hang while attempting to unmount the filesystem.
Signed-off-by: Sergio Lopez <slp@redhat.com>
There is no reason to give some special capabilities to the Rust version
of virtiofsd since it behaves slightly differently and does not require
neither DAC_OVERRIDE nor SYS_ADMIN.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
vhost_user_fs doesn't really support all vhost protocol features, just
MQ and SLAVE_REQ, so return that in protocol_features().
Signed-off-by: Sergio Lopez <slp@redhat.com>
Indirect descriptors is a virtio feature that allows the driver to
store a table of descriptors anywhere in memory, pointing to it from a
virtqueue ring's descriptor with a particular flag.
We can't seamlessly transition from an iterator over a conventional
descriptor chain to an indirect chain, so Queue users need to
explicitly support this feature by calling Queue::is_indirect() and
Queue::new_from_indirect().
Signed-off-by: Sergio Lopez <slp@redhat.com>
We import slave_fs_cache mod under vhost-user-slave feature control,
but not the self::slave_fs_cache::SlaveFsCacheReq import.
Signed-off-by: Eryu Guan <eguan@linux.alibaba.com>
We want to prevent from losing interrupts while they are masked. The
way they can be lost is due to the internals of how they are connected
through KVM. An eventfd is registered to a specific GSI, and then a
route is associated with this same GSI.
The current code adds/removes a route whenever a mask/unmask action
happens. Problem with this approach, KVM will consume the eventfd but
it won't be able to find an associated route and eventually it won't
be able to deliver the interrupt.
That's why this patch introduces a different way of masking/unmasking
the interrupts, simply by registering/unregistering the eventfd with the
GSI. This way, when the vector is masked, the eventfd is going to be
written but nothing will happen because KVM won't consume the event.
Whenever the unmask happens, the eventfd will be registered with a
specific GSI, and if there's some pending events, KVM will trigger them,
based on the route associated with the GSI.
Suggested-by: Liu Jiang <gerry@linux.alibaba.com>
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We should not assume the offset produced by ECAM is identical to the
CONFIG_ADDRESS register of legacy PCI port io enumeration.
Signed-off-by: Qiu Wenbo <qiuwenbo@phytium.com.cn>
This option improves the security of the guest by randomising the start
address of the kernel in physical memory. We should turn this on so as
to ensure all our functionality such as memory hotplug and kernel
loading works as this is an option used widely in production.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Recently, vhost_user_block gained the ability of actively polling the
queue, a feature that can be disabled with the poll_queue property.
This change adds this property to DiskConfig, so it can be used
through the "disk" argument.
For the moment, it can only be used when vhost_user=true, but this
will change once virtio-block gets the poll_queue feature too.
Fixes: #787
Signed-off-by: Sergio Lopez <slp@redhat.com>
Fix "readonly" and "wce" defaults in cloud-hypervisor.yaml to match
their respective defaults in config.rs:DiskConfig.
Signed-off-by: Sergio Lopez <slp@redhat.com>
This is a perfectly acceptable situation as it causes the backend to
exit because the VMM has closed the connection. This addresses the
rather ugly reporting of errors from the backend that appears
interleaved with the output from the VMM.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Return an error wen recvmsg() returns without a message using the
libc::ECONNRESET error so that the upper levels will correctly
interpret this as the connection being broken.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
It's missing a few knobs (readonly, vhost, wce) that should be exposed
through the rest API.
Fixes: #790
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The kernel does not adhere to the ACPI specification (probably to work
around broken hardware) and rather than busy looping after requesting an
ACPI reset it will attempt to reset by other mechanisms (such as i8042
reset.)
In order to trigger a reset the devices write to an EventFd (called
reset_evt.) This is used by the VMM to identify if a reset is requested
and make the VM reboot. As the reset_evt is part of the VMM and reused
for both the old and new VM it is possible for the newly booted VM to
immediately get reset as there is an old event sitting in the EventFd.
The simplest solution is to "drain" the reset_evt EventFd on reboot to
make sure that there is no spurious events in the EventFd.
Fixes: #783
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Now that vhost_user_backend and vm-virtio do support EVENT_IDX, use it
in vhost_user_block to reduce the number of notifications sent between
the driver and the device.
This is specially useful when using active polling on the virtqueue,
as it'll be implemented by a future patch.
This is a snapshot of kvm_stat while generating ~60K IOPS with fio on
the guest without EVENT_IDX:
Event Total %Total CurAvg/s
kvm_entry 393454 20.3 62494
kvm_exit 393446 20.3 62494
kvm_apic_accept_irq 378146 19.5 60268
kvm_msi_set_irq 369720 19.0 58881
kvm_fast_mmio 370497 19.1 58817
kvm_hv_timer_state 10197 0.5 1715
kvm_msr 8770 0.5 1443
kvm_wait_lapic_expire 7018 0.4 1118
kvm_apic 2768 0.1 538
kvm_pv_tlb_flush 2028 0.1 360
kvm_vcpu_wakeup 1453 0.1 278
kvm_apic_ipi 1384 0.1 269
kvm_fpu 1148 0.1 164
kvm_pio 574 0.0 82
kvm_userspace_exit 574 0.0 82
kvm_halt_poll_ns 24 0.0 3
And this is the snapshot while doing the same thing with EVENT_IDX:
Event Total %Total CurAvg/s
kvm_entry 35506 26.0 3873
kvm_exit 35499 26.0 3873
kvm_hv_timer_state 14740 10.8 1672
kvm_apic_accept_irq 13017 9.5 1438
kvm_msr 12845 9.4 1421
kvm_wait_lapic_expire 10422 7.6 1118
kvm_apic 3788 2.8 502
kvm_pv_tlb_flush 2708 2.0 340
kvm_vcpu_wakeup 1992 1.5 258
kvm_apic_ipi 1894 1.4 251
kvm_fpu 1476 1.1 164
kvm_pio 738 0.5 82
kvm_userspace_exit 738 0.5 82
kvm_msi_set_irq 701 0.5 69
kvm_fast_mmio 238 0.2 4
kvm_halt_poll_ns 50 0.0 1
kvm_ple_window_update 28 0.0 0
kvm_page_fault 4 0.0 0
It can be clearly appreciated how the number of vm exits per second,
specially the ones related to notifications (kvm_fast_mmio and
kvm_msi_set_irq) is drastically lower.
Signed-off-by: Sergio Lopez <slp@redhat.com>
Add helpers to Vring and VhostUserSlaveReqHandler for EVENT_IDX, so
consumers of this crate can make use of this feature.
Signed-off-by: Sergio Lopez <slp@redhat.com>
VIRTIO_RING_F_EVENT_IDX is a virtio feature that allows to avoid
device <-> driver notifications under some circunstances, most
notably when actively polling the queue.
This commit implements support for in in the vm-virtio
crate. Consumers of this crate will also need to add support for it by
exposing the feature and calling using update_avail_event() and
get_used_event() accordingly.
Signed-off-by: Sergio Lopez <slp@redhat.com>
Relying on the latest vm-memory version, including the freshly
introduced structure GuestMemoryAtomic, this patch replaces every
occurrence of Arc<ArcSwap<GuestMemoryMmap> with
GuestMemoryAtomic<GuestMemoryMmap>.
The point is to rely on the common RCU-like implementation from
vm-memory so that we don't have to do it from Cloud-Hypervisor.
Fixes#735
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The integration test test_memory_mergeable_on has been fairly unstable
for quite some time now. Because it can take some time for the VM to be
spawned and to be able to perform a correct measure of the PSS, this
commit simply increases the time before such measure is done.
This should return more accurate PSS results, which should help
stabilize the test.
Fixes#781
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that vhost_user_fs rust daemon supports virtiofs's dax mode, this adds
the two dax tests accordingly.
Signed-off-by: Liu Bo <bo.liu@linux.alibaba.com>
This adds the missing part of supporting virtiofs dax on the slave end,
that is, receiving a socket pair fd from the master end to set up a
communication channel for sending setupmapping & removemapping messages.
Signed-off-by: Liu Bo <bo.liu@linux.alibaba.com>
This introduces setupmapping and removemapping methods to server.rs,
passthrough.rs and filesystem.rs in order to support virtiofs dax mode
inside guest.
Since we don't really want the server.rs to know that it is dealing with
vhost-user specifically, this is making it more generic by adding a new
trait which has three functions map()/unmap()/sync() corresponding to
fs_slave_{map, unmap, sync}, server.rs will take anything that implements
the trait.
Signed-off-by: Liu Bo <bo.liu@linux.alibaba.com>
This introduces SlaveFsCacheReq which implements
VhostUserMasterReqHandler to handle map/unmap requests.
Signed-off-by: Liu Bo <bo.liu@linux.alibaba.com>
The unit tests are run from cargo test through multiple threads of the
same process. For this reason, all these threads share their file
descriptors (because that's how this works on Linux), which means that
any of them can close a file descriptor opened from another thread.
In the context of create_listener() and accept_connection() tests, they
can run concurrently and this generates some failure when the file
descriptor create_listener() is binding to is being closed from the
accept_connection() test.
In order to avoid such race condition, this patch simply removes the
part of the unit test performing an explicit and unsafe file descriptor
closure.
Fixes#759
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
If no socket is supplied when enabling "vhost_user=true" on "--disk"
follow the "exe" path in the /proc entry for this process and launch the
network backend (via the vmm_path field.)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The unit tests require some specific Linux capabilities and also to have
access to /dev/kvm device. This commit makes sure we enable only what's
necessary instead of blindly enable full priviliges with --privileged
option.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We need the host IPC for sharing eventfds with KVM, and the host network
for VFIO.
We also enforce the no-seccomp setting on the container, to overcome any
potential filtering set by our container's Ubuntu base.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
When a virtio-fs device is created with a dedicated shared region, by
default the region should be mapped as PROT_NONE so that no pages can be
faulted in.
It's only when the guest performs the mount of the virtiofs filesystem
that we can expect the VMM, on behalf of the backend, to perform some
new mappings in the reserved shared window, using PROT_READ and/or
PROT_WRITE.
Fixes#763
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
If no socket is supplied when enabling "vhost_user=true" on "--net"
follow the "exe" path in the /proc entry for this process and launch the
network backend (via the vmm_path field.)
Currently this only supports creating a new tap interface as the network
backend also only supports that.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
If the connection to the backend fails, sleep and retry up to a count.
This failure could happen because the backend is not yet ready to handle
the connection which is a common case when working with a spawned
backend.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
It is necessary to do this at the start of the VMM execution rather than
later as it must be done in the main thread in order to satisfy the
checks required by PTRACE_MODE_READ_FSCREDS (see proc(5) and
ptrace(2))
The alternative is to run as CAP_SYS_PTRACE but that has its
disadvantages.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This allows the VMM to explicitly shutdown devices as part of the VM
shutdown ahead of what Drop::drop() would do.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add a build-script to propagate the git commit hash to other crates at
compile time through environment variables, and display the hash along
with the '--version' option.
Fixes#729
Signed-off-by: Bo Chen <chen.bo@intel.com>
The placement of the explicit run of "test_vfio" meant it was run with
MMIO rather than PCI which meant it always failed.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Just updating the documentation regarding the pre-creation of a tap
interface supporting multiqueue, by adding what the cloud-hypervisor
--net parameter should look like.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By detecting if an existing tap interface supports multiqueue, we now
have the information to determine if the command line parameters
regarding the number of queues is correct.
Fixes#738
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to identify better the problem whenever an existing TAP is
provided, this patch enhances the error returned for an easier
diagnosis of the problem.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
test_vfio has been failing consistently on the CI so mark it with
a "#[ignore]" and then forceably build it again but ignore the build
result.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Implement the exit_event() method on the VhostUserBackend trait. It is
necessary to specify a custom exit event id in this case as the loop is
also used for handling activity on the tap file descriptors.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Implement the newly added exit_event() method on the VhostUserBackend
trait to allow the backend to provide an EventFD for triggering an exit
of the worker thread.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Rather than handling the KILL_EVENT in the event handler itself use the
newly added support in VhostUserBackend for providing a kill event
framework.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
All backends currently provide their own implementation for triggering
the worker thread to exit via an EventFd. Modify the VhostUserBackend
trait to allow a common implementation strategy that backends can use to
provide an EventFd (and optional id) that can be used to trigger the
worker to exit.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Check the return value from the worker thread by saving the thread
handle and waiting for it to return.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The main thread returns a Result with any errors from it. Although the
error from the join itself was being returned the real error from the
thread was being ignored so ensure that it is forwarded.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This reverts commit 4a1af7f63c.
This change erroneously ignored the return value for the result which
meant that requests to break out of the loop due to a kill event were
lost.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
All our tests must be run as root and thus the build directory is owned
by root after we run any of them.
Start another container to fix all permissions whenever we're done with
our tests.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
By default we will run as root inside the container, which means all the
build artifacts will be owned by root. That prevents us from properly
cleaning our build from an unprivileged host user.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
When running the docker container there is no interactivity needed so
don't pass "-ti" to "docker run"
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
To mitigate Azure slow disk IO, we mount /tmp on tmpfs.
This is a reproduction of our CI environment, as setup by the
Jenkinsfile.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Remove duplicated code across the different devices by handling
the virtio feature pages in VirtioDevice itself rather than
in the backends. This works as no virtio devices use feature
bits beyond 64-bits.
Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
If the ioctl syscall KVM_CREATE_VM gets interrupted while creating the
VM, it is expected that we should retry since EINTR should not be
considered a standard error.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Having the InterruptManager trait depend on an InterruptType forces
implementations into supporting potentially very different kind of
interrupts from the same code base. What we're defining through the
current, interrupt type based create_group() method is a need for having
different interrupt managers for different kind of interrupts.
By associating the InterruptManager trait to an interrupt group
configuration type, we create a cleaner design to support that need as
we're basically saying that one interrupt manager should have the single
responsibility of supporting one kind of interrupt (defined through its
configuration).
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
We create 2 different interrupt managers for separately handling
creation of legacy and MSI interrupt groups.
Doing so allows us to have a cleaner interrupt manager and IOAPIC
initialization path. It also prepares for an InterruptManager trait
design improvement where we remove the interrupt source type dependency
by associating an interrupt configuration type to the trait.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The existing code taking care of the epoll loop was too restrictive as
it was considering all errors the same. But in case the error is EINTR,
this means the syscall has been interrupted while waiting, and it should
be resumed to wait again.
This patch enforces the parsing of the returned error and prevent the
code from assuming EINTR should be handled as all other errors.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Check the rust formatting rather than just reformatting code on the CI
agent.
Also fix a formatting error that slipped in whilst the cargo fmt check
was not working correctly.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In order, among other things, to use the development CLI to run specific
integration tests. For example, to run only the memory_overhead
integration test:
./scripts/dev_cli.sh tests --integration -- memory_overhead
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
A previous version of this change attempted to avoid panicking by not
using .expect() when handling an error when attempting to write to the
log file. Unfortunately the macro eprintln!() that was used to replace
the .expect() also has the behaviour of panicking if stderr cannot be
used. Instead swallow the error completely as if writing to the log has
failed at logging time it is almost certainly the case that any message
about the log would also not be seen.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Since the vhost-user-blk binary will be removed and the newer
release will integrate this block backend into cloud-hypervisor
binary. The block backend code has been added num_queues cmdline
support, we need update multiple queues help info for this
block-backend in the cloud-hypervisor.
Signed-off-by: Yang Zhong <yang.zhong@intel.com>
We measure the memory overhead that the VMM process adds to the guest VM
and compare it with a maximum acceptable limit. The test is run against
a simple VM, running 1 vCPU and 512MB of RAM. Although this is not by
any mean a comprehensive VMM overhead measurement, it will allow us to
detect when and if any PR makes our code cross an arbitrary memory
overhead threshold.
Fixes: #64
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
As this can happen during the running of the VMM we should be very
careful not to panic() as that can lead to a thread being used by the VM
disappearing.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
A reference to the VmFd is stored on the AddressManager so it is not
necessary to pass in the VmInfo into all methods that need it as it can
be obtained from the AddressManager.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The DeviceManager has a reference to the MemoryManager so use that to
get the GuestMemoryMmap rather than the version stored in the VmInfo
struct.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Remove the use of vm_info in methods to get the config and instead use
the config stored on the DeviceManager itself.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Remove some in/out parameters and instead rely on them as members of the
&mut self parameter. This prepares the way to more easily store state on
the DeviceManager.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Remove some in/out parameters and instead rely on them as members of the
&mut self parameter. A follow-up commit will change the callee functions
that create the devices themselves.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Modify these functions to take an &mut self and become methods on
DeviceManager. This allows the removal of some in/out parameters and
leads the way to further refactoring and simplification.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The MemoryManager should only be included on the I/O bus when doing ACPI
builds as that is the only time it will be interrogated.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Currently the MemoryManager is only used on the ACPI code paths after
the DeviceManager has been created. This will change in a future commit
as part of the refactoring so for now always include it but name it with
underscore prefix to indicate it might not always be used.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Adding the num_queues parameter for vhost-user-blk backend, which
can enable MQ support in the backend.
This patch has enabled the MQ support from handle_event, and the
vhost-user-backend crate will enable multiple threads to call this
handle_event to handle read/write operations.
Signed-off-by: Yang Zhong <yang.zhong@intel.com>
The previous code only support one queue, and we need
to support MQ in vhost user block device. This patch
can work with SPDK with MQ setting.
Signed-off-by: Yang Zhong <yang.zhong@intel.com>
We don't need to force the cargo-audit install, we can check if it's
already available instead and install if it's not.
Also, since we now have workspaces properly setup, we can call directly
into cargo fmt and avoid calling into find magic incantation.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The script is a development tool that runs all commands in a dedicated
container. This allows for containerized, isolated and reproducible
builds and CI runs.
The script supports the following command:
* build: Build Cloud Hypervisor binaries (debug and release)
* build-container: Build the container used by the script
* tests: Run unit, cargo and integration tests
$ ./scripts/dev_cli.sh help
Cloud Hypervisor dev_cli.sh
Usage: dev_cli.sh <command> [<command args>]
Available commands:
build [--debug|--release] [-- [<cargo args>]]
Build the Cloud Hypervisor binaries.
--debug Build the debug binaries. This is the default.
--release Build the release binaries.
tests [--unit|--cargo|--all]
Run the Cloud Hypervisor tests.
--unit Run the unit tests.
--cargo Run the cargo tests.
--integration Run the integration tests.
--all Run all tests.
build-container [--type]
Build the Cloud Hypervisor container.
--dev Build dev container. This is the default.
help
Display this help message.
Fixes: #682Fixes: #684
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Because of the new set of patches related to virtio-iommu allowing only
for the topology to be described through virtio configuration, this
patch updates the kernel branch and the kernel configuration our CI
relies on.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that devices attached to the virtual IOMMU are described through
virtio configuration, there is no need for the DeviceManager to store
the list of IDs for all these devices. Instead, things are handled
locally when PCI devices are being added.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Instead of relying on the ACPI tables to describe the devices attached
to the virtual IOMMU, let's use the virtio topology, as the ACPI support
is getting deprecated.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the new structures previously introduced, the new topology
feature is being fully implemented through this commit. This allows
the description of the devices attached to the virtual IOMMU, which
is why a new function attach_devices() has been introduced. It gives
the virtual IOMMU device the full list of devices which must be attached
to it, letting the device share this information through its virtio
configuration.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The virtio-iommu device defines a new virtio feature allowing the
topology to be discovered fully through virtio configuration.
By topology, it means describing the devices attached to the virtual
IOMMU. This is currently managed through ACPI with IORT and VIOT table,
but this is another way of describing it.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The virtio capability VIRTIO_PCI_CAP_PCI_CFG is exposed through the
device's PCI config space the same way other virtio-pci capabilities
are exposed.
The main and important difference is that this specific capability is
designed as a way for the guest to access virtio capabilities without
mapping the PCI BAR. This is very rarely used, but it can be useful when
it is too early for the guest to be able to map the BARs.
One thing to note, this special feature MUST be implemented, based on
the virtio specification.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to anticipate the need to support more features related to the
access of a device's PCI config space, this commits changes the self
reference in the function read_config_register() to be mutable.
This also brings some more flexibility for any implementation of the
PciDevice trait.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Add a socket and vhost_user parameter to this option so that the same
configuration option can be used for both virtio-block and
vhost-user-block. For now it is necessary to specify both vhost_user
and socket parameters as auto activation is not yet implemented. The wce
parameter for supporting "Write Cache Enabling" is also added to the
disk configuration.
The original command line parameter is still supported for now and will
be removed in a future release.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add a socket and vhost_user parameter to this option so that the same
configuration option can be used for both virtio-net and vhost-user-net.
For now it is necessary to specify both vhost_user and socket parameters
as auto activation is not yet implemented. The original command line
parameter is still supported for now.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This commit introduces a clear definition of the virtio-fs
configuration structure, allowing vhost-user-fs device to
rely on it.
This makes the code more readable for developers.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit reuses the clear definition of the virtio-blk
configuration structure, allowing both vhost-user-blk and
virtio-blk devices to rely on it.
This makes the code more readable for developers.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit introduces a clear definition of the virtio-net
configuration structure, allowing both vhost-user-net and
virtio-net devices to rely on it.
This makes the code more readable for developers.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to reduce the amount of times VMs are being started through
integration tests, this commit consolidates very similar tests related
to virtio-blk into a single one.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Just add a new integration test to verify that multiqueue support is
correctly supported and that we can find the right amount of queues in
the guest.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit improves the existing virtio-blk implementation, allowing
for better I/O performance. The cost for the end user is to accept
allocating more vCPUs to the virtual machine, so that multiple I/O
threads can run in parallel.
One thing to notice, the amount of vCPUs must be egal or superior to the
amount of queues dedicated to the virtio-blk device.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The number of queues and the size of each queue were not configurable.
In anticipation for adding multiqueue support, this commit introduces
some new parameters to let the user decide about the number of queues
and the queue size.
Note that the default values for each of these parameters are identical
to the default values used for vhost-user-blk, that is 1 for the number
of queues and 128 for the queue size.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The trait bound and non trait bound virtio devices can use the same
inner implementation.
Also, the virtio pausable trait definiton can also be factorized.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Now that we have factorized the common virtio pausable implementation,
it's cleaner to have a dedicated macro for control queue devices rather
than overload the macro prototype.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
By adding an internal layer of abstraction (the hidden VirtioPausable
trait), we can factorize the virtio common code.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Now that we unified epoll_thread to potentially be a vector of threads,
it makes sense to make it a plural field.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Although only the block and net virtio devices can actually be multi
threaded (for now), handling them as special cases makes the code more
complex.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Devices like virtio-pmem and virtio-fs require some dedicated memory
region to be mapped. The memory mapping from the DeviceManager is being
replaced by the usage of MmapRegion from the vm-memory crate.
The unmap will happen automatically when the MmapRegion will be dropped,
which should happen when the DeviceManager gets dropped.
Fixes#240
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
When the CI runs in a brand new VM, there's no problem with the validity
of the images as they just got downloaded from the Azure bucket.
In case of a user who runs the CI locally, while doing some debug, he
might provision the images with cloudinit at some point, and later try
to run the CI based on these same images. What happens is that the CI
might randomly fail because the provisioning will not happen again as
it already happened.
This patch ensures the CI to fail early and show an error message to
notify the user about the validity of the images, based on their
sha1sum.
Fixes#112
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Move GED device reporting of required device type to scan into an MMIO
region rather than an I/O port.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Rather than have the MemoryManager device sit on the I/O bus allocate
space for MMIO and add it to the MMIO bus.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The way to get the BAR size is by writing all 1's to the BAR register in
the PCI config space. The mechanism was in place but the parameters were
swapped. The data buffer was provided with the actual offset, while the
offset was provided with the actual all 1's dword. We were effectively
trying to write the real offset at the offset 0xffffffff, which was
failing and resulting in the size being wrong.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Update the kernel build instructions to use the configuration and branch
that we test and develop against.
Fixes: #521
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This commit relies on the interrupt manager and the resulting interrupt
source group to abstract the knowledge about KVM and how interrupts are
updated and delivered.
This allows the entire "devices" crate to be freed from kvm_ioctls and
kvm_bindings dependencies.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The interrupt manager is passed to the IOAPIC creation, and the IOAPIC
now creates an InterruptSourceGroup for MSI interrupts based on it.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By introducing a new InterruptManager dedicated to the IOAPIC, we don't
have to solve the chicken and eggs problem about which of the
InterruptManager or the Ioapic should be created first. It's also
totally fine to have two interrupt manager instances as they both share
the same list of GSI routes and the same allocator.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Add explicit exclusions with --net-backend from --block-backend (and
vice-versa.) And also with "--kernel" as this is the option for "VM boot" that is never optional.
Ideally we would conflcit the backend arguments against the "vm-group"
however this does not work as it includes some arguments that have a
default value set and thus clap thinks those arguments are always
provided. Conflicting with "--kernel" is thus a reasonable compromise.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Split the basic launching functionality into its own function in the
newly added vhost_user_block crate.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Split the basic launching functionality into its own function in the
newly added vhost_user_net crate.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Extract the majority of the code that provides the vhost-user-block
backend into its own crate and port the binary to use it.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Extract the majority of the code that provides the vhost-user-net
backend into its own crate and port the binary to use it.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Generate the VM params until after the logging has been enabled. This
will make it possible to reuse the logging for vhost backends.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Previous commit added 'direct' support for virtio-blk to be able to
open files backing disk images with O_DIRECT, so add an integration
test for this new feature.
Signed-off-by: Sergio Lopez <slp@redhat.com>
vhost_user_blk already has it, so it's only fair to give it to
virtio-blk too. Extend DiskConfig with a 'direct' property, honor
it while opening the file backing the disk image, and pass it to
vm_virtio::RawFile.
Fixes#631
Signed-off-by: Sergio Lopez <slp@redhat.com>
Previous commit added 'readonly' support for virtio-blk to be able to
configure read-only devices, so add an integration test for this new
feature.
Signed-off-by: Sergio Lopez <slp@redhat.com>
vhost_user_blk already has it, so it's only fair to give it to
virtio-blk too. Extend DiskConfig with a 'readonly' properly, and pass
it to vm_virtio::Block.
Signed-off-by: Sergio Lopez <slp@redhat.com>
The build is run against "--all-features", "pci,acpi", "pci" and "mmio"
separately. The clippy validation must be run against the same set of
features in order to validate the code is correct.
Because of these new checks, this commit includes multiple fixes
related to the errors generated when manually running the checks.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Both InterruptDelivery and InterruptParameters can be removed from the
pci crate as they are not used anymore.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
There's no need for assign_irq() or assign_msix() functions from the
PciDevice trait, as we can see it's never used anywhere in the codebase.
That's why it's better to remove these methods from the trait, and
slightly adapt the existing code.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Since the InterruptManager is never stored into any structure, it should
be passed as a reference instead of being cloned.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit replaces the way legacy interrupts were handled with the
brand new implementation of the legacy InterruptSourceGroup for KVM.
Additionally, since it removes the last bit relying on the Interrupt
trait, the trait and its implementation can be removed from the
codebase.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit replaces the way legacy interrupts were handled with the
brand new implementation of the legacy InterruptSourceGroup for KVM.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit replaces the way legacy interrupts were handled with the
brand new implementation of the legacy InterruptSourceGroup for KVM.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Relying on the previous commits, the legacy interrupt implementation can
be completed. The IOAPIC handler is used to deliver the interrupt that
will be triggered through the trigger() method.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By having a reference to the IOAPIC, the KvmInterruptManager is going
to be able to initialize properly the legacy interrupt source group.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to be able to use the InterruptManager abstraction with
virtio-mmio devices, this commit introduces InterruptSourceGroup's
skeleton for legacy interrupts.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
When KvmInterruptManager initializes a new InterruptSourceGroup, it's
only for PCI_MSI_IRQ case that it needs to allocate the GSI and create a
new InterruptRoute. That's why this commit moves the general code into
the specific use case.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
When the base InterruptIndex is different from 0, the loop allocating
GSI and HashMap entries won't work as expected. The for loop needs to
start from base, but the limit must be base+count so that we allocate
a number of "count" entries.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to let the InterruptManager be shared across both PCI and MMIO
devices, this commit moves the initialization earlier in the code.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
After refactoring a common function is used to setup these slots and
that function takes care of allocating a new slot so it is not necessary
to reserve the initial region slots.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Now that KVM specific interrupts are handled through InterruptManager
trait implementation, the vm-virtio crate does not need to rely on
kvm_ioctls and kvm_bindings crates.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that KVM specific interrupts are handled through InterruptManager
trait implementation, the pci crate does not need to rely on kvm_ioctls
and kvm_bindings crates.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on all the previous changes, we can at this point replace the
entire interrupt management with the implementation of InterruptManager
and InterruptSourceGroup traits.
By using KvmInterruptManager from the DeviceManager, we can provide both
VirtioPciDevice and VfioPciDevice a way to pick the kind of
InterruptSourceGroup they want to create. Because they choose the type
of interrupt to be MSI/MSI-X, they will be given a MsiInterruptGroup.
Both MsixConfig and MsiConfig are responsible for the update of the GSI
routes, which is why, by passing the MsiInterruptGroup to them, they can
still perform the GSI route management without knowing implementation
details. That's where the InterruptSourceGroup is powerful, as it
provides a generic way to manage interrupt, no matter the type of
interrupt and no matter which hypervisor might be in use.
Once the full replacement has been achieved, both SystemAllocator and
KVM specific dependencies can be removed.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
After the skeleton of InterruptManager and InterruptSourceGroup traits
have been implemented, this new commit takes care of fully implementing
the content of KvmInterruptManager (InterruptManager trait) and
MsiInterruptGroup (InterruptSourceGroup).
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit introduces an empty implementation of both InterruptManager
and InterruptSourceGroup traits, as a proper basis for further
implementation.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
These new traits are meant to abstract the knowledge about the
hypervisor and the type of interrupt being used from the perspective
of the devices.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Thanks to the recently introduced function notifier() in the
VirtioInterrupt trait, all vhost-user devices can now bypass
listening onto an intermediate event fd as they can provide the
actual fd responsible for triggering the interrupt directly to
the vhost-user backend.
In case the notifier does not provide the event fd, the code falls
back onto the creation of an intermediate event fd it needs to listen
to, so that it can trigger the interrupt on behalf of the backend.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The point is to be able to retrieve directly the event fd related to
the interrupt, as this might optimize the way VirtioDevice devices are
implemented.
For instance, this can be used by vhost-user devices to provide
vhost-user backends directly with the event fd triggering the
interrupt related to a virtqueue.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Callbacks are not the most Rust idiomatic way of programming. The right
way is to use a Trait to provide multiple implementation of the same
interface.
Additionally, a Trait will allow for multiple functions to be defined
while using callbacks means that a new callback must be introduced for
each new function we want to add.
For these two reasons, the current commit modifies the existing
VirtioInterrupt callback into a Trait of the same name.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
At this point, both MSI and MSI-X handle the KVM GSI routing update,
which means the vfio crate does not have to deal with it anymore.
Therefore, several functions can be removed from the vfio-pci code, as
they are not needed anymore.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that MsiConfig has access to both KVM VmFd and the list of GSI
routes, the update of the KVM GSI routes can be directly done from
MsiConfig instead of specifically from the vfio-pci implementation.
By moving the KVM GSI routes update at the MsiConfig level, any PCI
device such as vfio-pci, virtio-pci, or any other emulated PCI device
can benefit from it, without having to implement it on their own.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The same way we have MsixConfig in charge of managing whatever relates
to MSI-X vectors, we need a MsiConfig structure to manage MSI vectors.
The MsiCap structure is still needed as a low level API, but it is now
part of the MsiConfig which oversees anything related to MSI.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to factorize one step further, we let MsixConfig perform the
interrupt enabling/disabling. This is done by registering/unregistering
the KVM irq_fds of all GSI routes related to this device.
And now that MsixConfig is in charge of the irq_fds, vfio-pci must rely
on it to retrieve them and provide them to the vfio driver.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that MsixConfig has access to the irq_fd descriptors associated with
each vector, it can directly write to it anytime it needs to trigger an
interrupt.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that MsixConfig has access to both KVM VmFd and the list of GSI
routes, the update of the KVM GSI routes can be directly done from
MsixConfig instead of specifically from the vfio-pci implementation.
By moving the KVM GSI routes update at the MsixConfig level, both
vfio-pci and virtio-pci (or any other emulated PCI device) can benefit
from it, without having to implement it on their own.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Because MsixConfig will be responsible for updating KVM GSI routes at
some point, it is necessary that it can access the list of routes
contained by gsi_msi_routes.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Because MsixConfig will be responsible for updating the KVM GSI routes
at some point, it must have access to the VmFd to invoke the KVM ioctl
KVM_SET_GSI_ROUTING.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The point here is to let MsixConfig take care of the GSI allocation,
which means the SystemAllocator must be passed from the vmm crate all
the way down to the pci crate.
Once this is done, the GSI allocation and irq_fd creation is performed
by MsixConfig directly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to anticipate the need for both msi.rs and msix.rs to rely on
some KVM utils and InterruptRoute structure to handle the update of the
KVM GSI routes, this commit adds these utilities directly to the pci
crate. So far, these were exclusively used by the vfio crate, which is
why there were located there.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Because we will need to share the same list of GSI routes across
multiple PCI devices (virtio-pci, VFIO), this commit moves the creation
of such list to a higher level location in the code.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This test is a variant of test_boot_vhost_user_blk(), named
test_boot_vhost_user_blk_direct(), that instances the vhost-user-blk
daemon with 'direct=true', to test this recently introduced feature
for opening files with O_DIRECT.
Signed-off-by: Sergio Lopez <slp@redhat.com>
Add missing WCE (write-cache enable) property support. This not only
an enhancement, but also a fix for a bug.
Right now, when vhost_user_blk uses a qcow2 image, it doesn't write
the QCOW2 metadata until the guest explicitly requests a flush. In
practice, this is equivalent to the write back semantic.
Without WCE, the guest assumes write through for the virtio_blk
device, and doesn't send those flush requests. Adding support for WCE,
and enabling it by default, we ensure the guest does send said
requests.
Supporting "WCE = false" would require updating our qcow2
implementation to ensure that, when required, it honors the write
through semantics by not deferring the updates to QCOW2 metadata.
Signed-off-by: Sergio Lopez <slp@redhat.com>
Add support for opening the disk images with O_DIRECT. This allows
bypassing the host's file system cache, which is useful to avoid
polluting its cache and for better data integrity.
This mode of operation can be enabled by adding the "direct=<bool>"
parameter to the "backend" argument:
./target/debug/vhost_user_blk --backend image=test.raw,sock=/tmp/vhostblk,direct=true
The "direct" parameter defaults to "false", to preserve the original
behavior.
Signed-off-by: Sergio Lopez <slp@redhat.com>
Use RawFile as backend instead of File. This allows us to abstract
the access to the actual image with a specialized layer, so we have a
place where we can deal with the low-level peculiarities.
Signed-off-by: Sergio Lopez <slp@redhat.com>
Doing I/O on an image opened with O_DIRECT requires to adhere to
certain restrictions, requiring the following elements to be aligned:
- Address of the source/destination memory buffer.
- File offset.
- Length of the data to be read/written.
The actual alignment value depends on various elements, and according
to open(2) "(...) there is currently no filesystem-independent
interface for an application to discover these restrictions (...)".
To discover such value, we iterate through a list of alignments
(currently, 512 and 4096) calling pread() with each one and checking
if the operation succeeded.
We also extend RawFile so it can be used as a backend for QcowFile,
so the later can be easily adapted to support O_DIRECT too.
Signed-off-by: Sergio Lopez <slp@redhat.com>
Update queue number with 4 to verify if vhost-user-net device
and backend could work well with multiple queue.
Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
There are two new options num_queues and queue_size defined for
virtio-net, add them in test_valid_vm_config_net which is used
to validate that both the CLI and the OpenAPI will generate the
same configuration.
Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
Update the common part in net_util.rs under vm-virtio to add mq
support, meanwhile enable mq for virtio-net device, vhost-user-net
device and vhost-user-net backend. Multiple threads will be created,
one thread will be responsible to handle one queue pair separately.
To gain the better performance, it requires to have the same amount
of vcpus as queue pair numbers defined for the net device, due to
the cpu affinity.
Multiple thread support is not added for vhost-user-net backend
currently, it will be added in future.
Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
Add num_queues and queue_size for virtio-net device to make them configurable,
while add the associated options in command line.
Update cloud-hypervisor.yaml with the new options for NetConfig.
Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
Add support to allow VMMs to open the same tap device many times, it will
create multiple file descriptors meanwhile.
Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
Current guest kernel will check the oneline cpu count, in principle,
if the online cpu count is not smaller than the number of queue pairs
VMM reported, the net packets could be put/get to all the virtqueues,
otherwise, only the number of queue pairs that match the oneline cpu
count will have packets work with. guest kernel will send command
through control queue to tell VMMs the actual queue pair numbers which
it could currently play with. Add mq process in control queue handling
to get the queue pair number, VMM will verify if it is in a valid range,
nothing else but this.
Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
While feature VIRTIO_NET_F_CTRL_VQ is negotiated, control queue
will exits besides the Tx/Rx virtqueues, an epoll handler should
be started to monitor and handle the control queue event.
Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
As virtio spec 1.1 said, the driver uses the control queue
to send commands to manipulate various features of the devices,
such as VIRTIO_NET_F_MQ which is required by multiple queue
support. Here add the control queue handling process.
Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
Since the common parts are put into net_util.rs under vm-virtio,
refactoring code for virtio-net device, vhost-user-net device
and backend to shrink the code size and improve readability
meanwhile.
Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
There are some common logic shared among virtio-net device, vhost-user-net
device and vhost-user-net backend, abstract those parts into net_util.rs
to improve code maintainability and readability.
Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
According to virtio spec, for used buffer notifications, if
MSI-X capability is enabled, and queue msix vector is
VIRTIO_MSI_NO_VECTOR 0xffff, the device must not deliver an
interrupt for that virtqueue.
Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
Use independent bits for storing whether there is a CPU or memory device
changed when reporting changes via ACPI GED interrupt. This prevents a
later notification squashing an earlier one and ensure that hotplugging
both CPU and memory at the same time succeeds.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
If a new amount of RAM is requested in the VmResize command try and
hotplug if it an increase (MemoryManager::Resize() silently ignores
decreases.)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
If there is a GED interrupt and the field indicates that the memory
device has changed triggers a scan of the memory devices.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Generate and expose the DSDT table entries required to support memory
hotplug. The AML methods call into the MemoryManager via I/O ports
exposed as fields.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Expose the details of hotplug RAM slots via an I/O port. This will be
consumed by the ACPI DSDT tables to report the hotplug memory details to
the guest.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add a "resize()" method on MemoryManager which will create a new memory
allocation based on the difference between the desired RAM amount and
the amount already in use. After allocating the added RAM using the same
backing method as the boot RAM store the details in a vector and update
the KVM map and create a new GuestMemoryMmap and replace all the users.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
For now the new memory size is only used after a reboot but support for
hotplugging memory will be added in a later commit.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When the value is read from the I/O port via the ACPI AML functions to
determine what has been triggered the notifiction value is reset
preventing a second read from exposing the value. If we need support
multiple types of GED notification (such as memory hotplug) then we
should avoid reading the value multiple times.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
These fields allow you to expose the component part of an existing
buffer, such as a resource template as a new field which is required to
expose the memory CRS details.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This specifies how much address space should be reserved for hotplugging
of RAM. This space is reserved by adding move the start of the device
area by the desired amount.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In order to be able to support resizing either vCPUs or memory or both
make the fields in the resize command optional.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Make the GuestMemoryMmap from a Vec<Arc<GuestRegionMmap>> by using this
method we can persist a set of regions in the MemoryManager and then
extend this set with a newly created region. Ultimately that will allow
the hotplug of memory.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
If neither PCI or MMIO are built in, we should not bother creating any
virtio devices at all.
When building a minimal VMM made of a kernel with an initramfs and a
serial console, the RNG virtio device is still created even though there
is no way it can ever get probed.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The current virtio-fs-dev branch from the official repo is breaking the
build on our Azure VMs because the glibc wrapper for renameat2() syscall
does not exist. This branch fixes the problem while the maintainers are
fixing the issue upstream.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
With the latest version of virtiofsd, some changes have been made
to the socket path parameter. This needs to be updated in the cloud
hypervisor codebase, so that our CI keeps running correctly.
Fixes#611
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The From and Display traits were not handling some of the enum
definitions. We no longer have a default case for Display so any future
misses will fail at build time.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Everytime we use VFIO with cloud-hypervisor, we get the following error:
ERROR:vfio/src/vfio_device.rs:440 -- Could not get region #8 info
But this is not an error per se, and should be considered as a simple
warning.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The current code was always considering 0xffffffff being written to the
register as a sign it was expecting to get the size, hence the BAR
reprogramming detection was stating this case was not a reprogramming
case.
Problem is, if the value 0xffffffff is directed at a 64bits BAR, this
might be the high or low part of a 64bits address which is meant to be
the new address of the BAR, which means we would miss the detection of
the BAR being reprogrammed here.
This commit improves the code using finer granularity checks in order to
detect this corner case correctly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The expansion ROM BAR reprogramming was being triggered for the wrong
reason and was causing the following error to be reported:
ERROR:pci/src/bus.rs:207 -- Failed moving device BAR: failed allocating
new 32 bits MMIO range
Following the PCI specification, here is what is defined:
Device independent configuration software can determine how much address
space the device requires by writing a value of all 1's to the address
portion of the register and then reading the value back.
This means we cannot expect 0xffffffff to be written, as the address
portion corresponds to the bits 31-11. That's why whenever the size of
this special BAR is being asked for, the value being written is
0xfffff800.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The way the code is currently implemented, only by writing to STDIN a
user can trigger some input to reach the VM through virtio-console. But
in case, there were not enough virtio descriptors to process what was
retrieved from STDIN, the remaining bits would be transferred only if
STDIN was triggered again. The missing part is that when some
descriptors are made available from the guest, the virtio-console device
should try to send any possible remaining bits.
By triggering the function process_input_queue() whenever the guest
notifies the host that some new descriptors are ready for the receive
queue, this patch allows to fill the implementation void that was left.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In case the virtio descriptor is pulled out of the Queue iterator, it
is important to fill it and tag it as used. This is already done from
the successful code path, but in case there's an error during the
filling, we should make sure to put the descriptor back in the list of
available descriptors. This way, when the error occurs, we don't loose
a descriptor, and it could be used later.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The existing code was a bit too complex and it was introducing a bug
when trying to paste long lines directly to the console. By simplifying
the code, and by doing proper usage of the drain() function, the bug is
fixed by this commit.
Here is the similar output one could have gotten from time to time, when
pasting important amounts of bytes:
ERROR:vm-virtio/src/console.rs:104 -- Failed to write slice:
InvalidGuestAddress(GuestAddress(1040617472))
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The cache= option from the virtiofsd backend is meant to be used
depending on how the user wants to use the guest page cache. For most
use cases, we want to bypass the guest page cache to reduce the guest
memory footprint, which is why "cache=none" should be the default.
"cache=always" is the other option when the user wants to do something
specific by using the guest page cache, but should not be the default
anymore.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The virtio-fs messages coming from the slave can contain multiple
mappings (up to 8) through one single request. By implementing such
feature, the virtio-fs implementation of cloud-hypervisor is optimal and
fully functional as it resolves a bug that was seen when running fio
testing without this patch.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
With latest kernel, virtio-fs mount command has been simplified, and
this needs to be applied everywhere in our tests and documentation.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By implementing this virtio feature, we let the virtio-iommu driver call
the device backend so that it can probe each device that gets attached.
Through this probing, the device provides a range of reserved memory
related to MSI. This is mandatory for x86 architecture as we want to
avoid the default MSI range assigned by the virtio-iommu driver if no
range is provided at all. The default range is 0x8000000-0x80FFFFF but
it only makes sense for ARM architectures.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Because virtio-iommu is still evolving (as it's only partly upstream),
some pieces like the ACPI declaration of the different nodes and devices
attached to the virtual IOMMU are changing.
This patch introduces a new ACPI table called VIOT, standing as the high
level table overseeing the IORT table and associated subtables.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to validate the newly created virtio-fs-virtio-iommu-5.5-rc1
branch is not breaking anything in our CI, we must try it as a temporary
branch before we can override virtio-fs-virtio-iommu.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Make all the crates members of the workspace so that "cargo test
--workspace" will find them all and test them with the features enabled
that we use.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
If we use the workspace based testing methodology then we start testing
some code inside vhost_rs that is broken.
See #576
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The following commit broke this unit test:
"""
vmm: Convert virtio devices to Arc<Mutex<T>>
Migratable devices can be virtio or legacy devices.
In any case, they can potentially be tracked through one of the IO bus
as an Arc<Mutex<dyn BusDevice>>. In order for the DeviceManager to also
keep track of such devices as Migratable trait objects, they must be
shared as mutable atomic references, i.e. Arc<Mutex<T>>. That forces all
Migratable objects to be tracked as Arc<Mutex<dyn Migratable>>.
Virtio devices are typically migratable, and thus for them to be
referenced by the DeviceManager, they now should be built as
Arc<Mutex<VirtioDevice>>.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
"""
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This allows us to change the memory map that is being used by the
devices via an atomic swap (by replacing the map with another one). The
ArcSwap provides the mechanism for atomically swapping from to another
whilst still giving good read performace. It is inside an Arc so that we
can use a single ArcSwap for all users.
Not covered by this change is replacing the GuestMemoryMmap itself.
This change also removes some vertical whitespace from use blocks in the
files that this commit also changed. Vertical whitespace was being used
inconsistently and broke rustfmt's behaviour of ordering the imports as
it would only do it within the block.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This function will be useful for other parts of the VMM that also
estabilish their own mappings.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This removes the need to handle a mutable integer and also centralises
the allocation of these slot numbers.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The memory manager is responsible for setting up the guest memory and in
the long term will also handle addition of guest memory.
In this commit move code for creating the backing memory and populating
the allocator into the new implementation trying to make as minimal
changes to other code as possible.
Follow on commits will further reduce some of the duplicated code.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Because we don't always reach the expected footprint improvements with
KSM, let's review the numbers. By reducing the expectations and
increasing the amount of pages to scan, this should stabilize the CI.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
To reflect updated clippy rules:
error: `if` chain can be rewritten with `match`
--> vmm/src/device_manager.rs:1508:25
|
1508 | / if ret > 0 {
1509 | | debug!("MSI message successfully delivered");
1510 | | } else if ret == 0 {
1511 | | warn!("failed to deliver MSI message, blocked by guest");
1512 | | }
| |_________________________^
|
= note: `-D clippy::comparison-chain` implied by `-D warnings`
= help: Consider rewriting the `if` chain to use `cmp` and `match`.
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#comparison_chain
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Address updated clippy errors:
error: redundant clone
--> vmm/src/device_manager.rs:699:32
|
699 | .insert(acpi_device.clone(), 0x3c0, 0x4)
| ^^^^^^^^ help: remove this
|
= note: `-D clippy::redundant-clone` implied by `-D warnings`
note: this value is dropped without further use
--> vmm/src/device_manager.rs:699:21
|
699 | .insert(acpi_device.clone(), 0x3c0, 0x4)
| ^^^^^^^^^^^
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#redundant_clone
error: redundant clone
--> vmm/src/device_manager.rs:737:26
|
737 | .insert(i8042.clone(), 0x61, 0x4)
| ^^^^^^^^ help: remove this
|
note: this value is dropped without further use
--> vmm/src/device_manager.rs:737:21
|
737 | .insert(i8042.clone(), 0x61, 0x4)
| ^^^^^
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#redundant_clone
error: redundant clone
--> vmm/src/device_manager.rs:754:29
|
754 | .insert(cmos.clone(), 0x70, 0x2)
| ^^^^^^^^ help: remove this
|
note: this value is dropped without further use
--> vmm/src/device_manager.rs:754:25
|
754 | .insert(cmos.clone(), 0x70, 0x2)
| ^^^^
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#redundant_clone
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
We use the version that's included in the linux-loader crate. This old
version was also generating build errors after updating to the new
clippy:
e.g.:
error: unsafe function's docs miss `# Safety` section
--> arch_gen/src/x86/bootparam.rs:23:5
|
23 | / pub unsafe fn as_ptr(&self) -> *const T {
24 | | ::std::mem::transmute(self)
25 | | }
| |_____^
|
= note: `-D clippy::missing-safety-doc` implied by `-D warnings`
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#missing_safety_doc
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This patch has been cherry-picked from the Firecracker tree. The
reference commit is 1db04ccc69862f30b7814f30024d112d1b86b80e.
Changed the host-initiated vsock connection protocol to include a
trivial handshake.
The new protocol looks like this:
- [host] CONNECT <port><LF>
- [guest/success] OK <assigned_host_port><LF>
On connection failure, the host host connection is reset without any
accompanying message, as before.
This allows host software to more easily detect connection failures, for
instance when attempting to connect to a guest server that may have not
yet started listening for client connections.
Signed-off-by: Dan Horobeanu <dhr@amazon.com>
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This patch is to check if block device is readonly
when backend set readonly=true.
The lsblk command can show the RO value in the guest.
Signed-off-by: Yang Zhong <yang.zhong@intel.com>
The current backend only support rw, and we also need
add readonly support.
The new command:
vhost_user_blk \
--backend "image=/home/test.img, \
sock=/home/path/vhost.socket, \
readonly=true"
Signed-off-by: Yang Zhong <yang.zhong@intel.com>
When the running OS has been told that a CPU should be removed it will
shutdown the CPU and then signal to the hypervisor via the "_EJ0" method
on the device that ultimately writes into an I/O port than the vCPU
should be shutdown. Upon notification the hypervisor signals to the
individual thread that it should shutdown and waits for that thread to
end.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Allow the resizing of the number of vCPUs to less than the current
active vCPUs. This does not currently remove them from the system but
the kernel will take them offline.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When we add a vCPU set an "inserting" boolean that is exposed as an ACPI
field that will be checked for and reset when the ACPI GED notification
for CPU devices happens.
This change is a precursor for CPU unplug.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Continue to notify on all vCPUs but instead separate the notification
functionality into two methods, CSCN that walks through all the CPUs
and CTFY which notifies based on the numerical CPU id. This is an
interim step towards only notifying on changed CPUs and ultimately CPU
removal.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The goal here is to ensure that CLI and OpenAPI both behave as closely
as possible, and also that they behave as expected.
Leveraging the reorganization of the code, we can now compare two
VmConfig structures generated from one CLI entry on one side, and from
an OpenAPI entry (JSON payload) on the other side.
Fixes#535
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In anticipation for the writing of unit tests comparing two VmConfig
structures, this commit derives the PartialEq trait for VmConfig and
all embedded structures.
This patch also derives the Debug trait for the same set of structures
so that we can print them to facilitate debugging.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The OpenAPI should not have to provide a command line since the CLI
considers the command line as an empty string if nothing is provided.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This brings more modularity to the code, which will be helpful when we
will later test the CLI and OpenAPI generate the same VmConfig output.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This brings more modularity to the code, which will be helpful when we
will later test the CLI and OpenAPI generate the same VmConfig output.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This brings more modularity to the code, which will be helpful when we
will later test the CLI and OpenAPI generate the same VmConfig output.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
On our CI the /tmp filesystem is mounted as tmpfs and this is the
location where the test disk images are located. When the CI worker
nodes have less memory and fewer CPUs the tmpfs fills up as the tests
run in parallel.
Introduce a mechanism to reduce the parallelism of the tests based on
starvation of the tmpfs disk availability.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The vsock packets that we're building are resolving guest addresses to
host ones and use the latter as raw pointers.
If the corresponding guest mapped buffer spans across several regions in
the guest, they will do so in the host as well. Since we have no
guarantees that host regions are contiguous, it may lead the VMM into
trying to access memory outside of its memory space.
For now we fix that by ensuring that the guest buffers do not span
across several regions. If they do, we error out.
Ideally, we should enhance the rust-vmm memory model to support safe
acces across host regions.
Fixes CVE-2019-18960
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Highlight that we support Ubuntu Bionic and Eoan as well as Clear Linux
and update the Clear Linux versions referenced.
Also update the firmware URL to point to the latest version.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Virtio-fs maintainers recently updated the virtiofsd daemon through
their official branch on Gitlab. It includes fixes that were needed for
cloud-hypervisor to work correctly with it.
Jenkinsfile needs to be updated since the virtiofsd build requires both
libseccomp and libcap-ng to be present on the system.
One thing to notice, because the latest branch introduced a change
regarding libfuse behavior, the counterpart patch has been added to the
custom kernel branch "virtio-fs-virtio-iommu".
Fixes#536
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Since the Snapshotable placeholder and Migratable traits are provided as
well, the DeviceManager object and all its objects are now Migratable.
All Migratable devices are tracked as Arc<Mutex<dyn Migratable>>
references.
Keeping track of all migratable devices allows for implementing the
Migratable trait for the DeviceManager structure, making the whole
device model potentially migratable.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Due to the amount of code currently duplicated across vhost-user devices,
the stats for this commit is on the large side but it's mostly more
duplicated code, unfortunately.
Migratable and Snapshotable placeholder implementations are provided as
well, making all vhost-user devices Migratable.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Due to the amount of code currently duplicated across virtio devices,
the stats for this commit is on the large side but it's mostly more
duplicated code, unfortunately.
Migratable and Snapshotable placeholder implementations are provided as
well, making all virtio devices Migratable.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Migratable devices can be virtio or legacy devices.
In any case, they can potentially be tracked through one of the IO bus
as an Arc<Mutex<dyn BusDevice>>. In order for the DeviceManager to also
keep track of such devices as Migratable trait objects, they must be
shared as mutable atomic references, i.e. Arc<Mutex<T>>. That forces all
Migratable objects to be tracked as Arc<Mutex<dyn Migratable>>.
Virtio devices are typically migratable, and thus for them to be
referenced by the DeviceManager, they now should be built as
Arc<Mutex<VirtioDevice>>.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The anyhow crate generates some incorrectly indented code from its
build.rs code. We don't want to run cargo fmt on this code.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The Migratable trait groups all expected capabilities of devices and
components that can be migrated.
For a component to be migrated, it must be able to pause and resume.
Once paused, it should be able to provide a snapshot of itself. It
should also be able to restore itself from a snaphot.
As a consequence, the Migratable trait will be split between the
Pausable and the Snapshotable traits. This commit only adds the
Pausable one.
All migratable devices will be tracked from the DeviceManager.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The FsConfig structure has been recently adjusted so that the default
value matches between OpenAPI and CLI. Unfortunately, with the current
description, there is no way from the OpenAPI to describe a cache_size
value "None", so that DAX does not get enabled. Usually, using a Rust
"Option" works because the default value is None. But in this case, the
default value is Some(8G), which means we cannot describe a None.
This commit tackles the problem, introducing an explicit parameter
"dax", and leaving "cache_size" as a simple u64 integer.
This way, the default value is dax=true and cache_size=8G, but it lets
the opportunity to disable DAX entirely with dax=false, which will
simply ignore the cache_size value.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to let the CLI and the HTTP API behave the same regarding the
VhostUserBlkConfig structure, this patch defines some default values
for num_queues, queue_size and wce.
num_queues is 1, queue_size is 128 and wce is true.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to let the CLI and the HTTP API behave the same regarding the
VhostUserNetConfig structure, this patch defines some default values
for num_queues, queue_size and mac.
num_queues is 2 since that's a pair of TX/RX queues, queue_size is 256
and mac is a randomly generated value.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We want to set different default configurations for vhost-user-net and
vhost-user-blk, which is the reason why the common part corresponding to
the number of queues and the queue size cannot be embedded.
This prepares for the following commit, matching API and CLI behaviors.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
A simple patch making sure the field "file" is provisioned with the same
default value through CLI and OpenAPI.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Just making sure we have a serde default for the field "file" since it
is not a required field in the OpenAPI definition.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
All structures match between the OpenAPI definition and the internal
configuration code, that's why CpuConfig is being renamed into
CpusConfig.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By default, and in order to avoid falling into the legacy CLI usage, the
CPU argument should at least include "boot=" to define the number of
CPUs.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The signal handling for vCPU signals has changed in the latest release
so switch to the new API.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
We need to validate that OpenAPI YAML definition is not broken by each
and every pull request. The easiest way is to rely on the Docker image
provided by OpenAPITools, as it allows us to validate the definition
with one simple command.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that Cargo tests are being run on the worker node directly, we can
leave the bare minimum in Travis, which is taking care of the release
deployment.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Because the resources on the amount of worker nodes we can have access
to through Travis is limited, we offload the burden of running all tests
related to Cargo inside the Azure VM directly.
This will have the positive effect of stopping the build very early in
case something goes wrong during the Cargo testing.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to let the CLI and the HTTP API behave the same regarding the
FsConfig structure, this patch defines some default values for
num_queues, queue_size and the cache_size.
num_queues is set to 1, queue_size is set to 1024, and cache_size is set
to Some(8G) which means that DAX is enabled by default with a shared
region of 8GiB.
Fixes#508
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to let the CLI and the HTTP API behave the same regarding the
NetConfig structure, this patch defines some default values for tap, ip,
mask, mac and iommu.
tap is None, ip is 192.168.249.1, mask is 255.255.255.0, mac is a
randomly generated value, and iommu is false.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Following the recent addition of CONFIG_EXPERT=y, the kernel config now
asks for several options through interactive shell. This is not
convenient when locally running the integration tests script as the
shell asks for input.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that the GED device does not use a hardcoded IRQ number the starting
IRQ number can be restored (needed for the hardcoded serial port IRQ.)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Remove the previously hardcoded IRQ number used for the GED device.
Instead allocate the IRQ using the allocator and use that value in the
definition in the ACPI device.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Move the code for handling the creation of the DSDT entries for devices
into the DeviceManager.
This will make it easier to handle device hotplug and also in the future
remove some hardcoded ACPI constants.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Move the code for generating the MADT (APIC) table and the DSDT
generation for CPU related functionality into the CpuManager.
There is no functional change just code rearrangement.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When consumer of the HTTP API try to interact with cloud-hypervisor,
they have to provide the equivalent of the config structure related to
each component they need. Problem is, the Rust enum type "Option" cannot
be obtained from the OpenAPI YAML definition.
This patch intends to fix this inconsistency between what is possible
through the CLI and what's possible through the HTTP API by using simple
types bool and int64 instead of Option<u64>.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to validate that multiple devices can be passed through and
they are still fully functional, this patch extends the existing VFIO
test to pass a second virtio-net device, and verifies that both
interfaces are functional by ssh'ing into each network interface.
Fixes#503
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Previously the device setup code assumed that if no IOAPIC was passed in
then the device should be added to the kernel irqchip. As an earlier
change meant that there was always a userspace IOAPIC this kernel based
code can be removed.
The accessor still returns an Option type to leave scope for
implementing a situation without an IOAPIC (no serial or GED device).
This change does not add support no-IOAPIC mode as the original code did
not either.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Update the configuration after a resize to ensure that after a reboot
the added vCPUs are preserved.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This was causing issues when the kernel was trying to reset the
interrupt and making the reboot fail.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The tty mode remains raw mode when cloud-hypervisor is terminted by
SIGTERM or SIGINT. The terminal is unusable due to echoing is
disabled which is really annoying.
Signed-off-by: Qiu Wenbo <qiuwenbo@phytium.com.cn>
This commit ensures device's PCI config space is being written after
MSI/MSI-X interrupts have been enabled/disabled. In case of MSI, when
the interrupts are enabled through VFIO (using VFIO_DEVICE_SET_IRQS),
the MSI Enable bit in the MSI capability structure found in the PCI
config space is disabled by default. That's why when the guest is
enabling this bit, we first need to enable the MSI interrupts with
VFIO through VFIO_DEVICE_SET_IRQS ioctl, and only after we can write
to the device region to update the MSI Enable bit.
Fixes#460
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The comments of vfio kernel module said that individual subindex
interrupts can be disabled using the -1 value for DATA_EVENTFD or
the index can be disabled as a whole with:
flags = (DATA_NONE|ACTION_TRIGGER), count = 0.
Signed-off-by: Wu Zongyong <wuzongyong@linux.alibaba.com>
CC: Liu Jiang <gerry@linux.alibaba.com>
Some physical address bits may become reserved in page table when SME
is enabled on AMD platform. Guest will trigger a reserved bit
violation page fault in this case due to write these reserved bits to 1
in page table. We need reduce the reserved bits to get the right
physical address range.
Signed-off-by: Qiu Wenbo <qiuwenbo@phytium.com.cn>
The KVM_SET_GSI_ROUTING ioctl is very simple, it overrides the previous
routes configuration with the new ones being applied. This means the
caller, in this case cloud-hypervisor, needs to maintain the list of all
interrupts which needs to be active at all times. This allows to
correctly support multiple devices to be passed through the VM and being
functional at the same time.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to improve the existing VFIO code, this patch registers the
eventfds used to trigger KVM interrupts only when the interrupts are
enabled, and unregisters them when interrupts are disabled.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
If the MSG_CTL is being written from a 32 bits write access, the offset
won't be 0x2, but 0x0 instead. That's simply because 32 bits access have
to be aligned on each double word.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The documentation was out of date since the URL to download virtiofsd
binary was not valid anymore. Instead, the updated documentation now
describe how to build virtiofsd from source.
Fixes#491
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Call the "CTFY" method that will itself call Notify() on the CPU objects
in the ACPI namespace.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add ability to notify via the GED device that there is some new hotplug
activity. This will be used by the CpuManager (and later DeviceManager
itself) to notify of new hotplug activity.
Currently it has a hardcoded IRQ of 5 as the ACPI tables also need to
refer to this IRQ and the IRQ allocation does not permit the allocation
of specific IRQs.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This device provides the ability to notify the kernel via interrupt that
there is some new hotplug activity. The type is determined by reading
from the I/O port.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Currently only increasing the number of vCPUs is supported but in the
future it will be extended.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When configuring a processor after boot as a hotplug CPU we only
configure a subset of the CPU state. In particular we should not
configure the FPU, segment registers (or reconfigure the paging which is
a side-effect of that) nor the main registers. Achieve this by making
the function take an Option type for the start address.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Refactor the vCPU thread starting so that there is the possibility to
bring on extra vCPU threads.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Currently this just holds the thread handle but will be enlarged to
encompass details such as whether the vCPU is currently being inserted
or ejected.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The easiest way to detect if the kernel is willing to accept hotplug
vCPUs is to check the dmesg output.
Switch the test to bionic as the Clear Cloud image lacks "dmesg."
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The MADT table contains the details of all the potential vCPUs and
whether they are present at boot (as indicated by the flags field.)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When initialising the ACPI tables and configuring the VM use the new
accessor on the CpuManager to get the number of boot vCPUs.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The way in which offsets are currently use in memory regions is
derived from QEMU's contrib/libvhost-user, but while this one works
mainly by translating vmm va's to local va's, vm-memory expects us to
use proper guest addresses and thus, define memory regions that
actually match the guest's memory disposition.
With this change, we create the memory regions with the proper length
and offsets, extend AddrMapping to store the guest physical address,
and use the latter instead of offset in vmm_va_to_gpa().
Signed-off-by: Sergio Lopez <slp@redhat.com>
Since the kvm crates now depend on vmm-sys-util, the bump must be
atomic.
The kvm-bindings and ioctls 0.2.0 and 0.4.0 crates come with a few API
changes, one of them being the use of a kvm_ioctls specific error type.
Porting our code to that type makes for a fairly large diff stat.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Reorder the build commands so that more build assets in the target/
build directory can be reused. This will reduce the build time.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Introduce method unmap_mmio_regions to unmap all regions mapped to host.
This patch eliminate the error message "Could not unset container".
Signed-off-by: Wu Zongyong <wuzongyong@linux.alibaba.com>
context ID on vsock man defines a 32-bits value, openapi default integer
is a signed 32-bits value.
This could lead to miss one bit during castings for typed client
implmentations. Lets increase the range of valid values by requesting an
int64.
Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
This reverts commit 66fde245b3.
The commit broke the VFIO support for MSI. Issue needs to be
investigated but in the meantime, it is safer to fix the codebase.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The setcap tool is unusual in its behaviour in that it will process each
parameter separately and abort if one of the parameters is invalid (e.g.
after a symlink.) But any previous parameters will be correctly
processed. This means that depending on the generated test binary name
an invalid entry might occur before it and thus abort the setcap.
Fix to only setcap on the test binary by excluding the other files.
Because the binary name is based on a hash the PR that introduced this
version worked but once merged the hash changed and broke the build.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In order to validate the new virtio-fs daemon written in Rust is
behaving correctly, a new integration test has been added. Important to
note that for now, only a test with cache=none and dax=off can be added
since the daemon does not support shared memory region yet.
The long term goal being to replace virtiofsd with vhost_user_daemon
once it will reach parity regarding the supported features.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Because the vhost_user_backend crate needs some changes to support
moving the process to a different mount namespace and perform a pivot
root, it is not possible to change '/' to the given shared directory.
This commit, as a temporary measure, let the code point at the given
shared directory.
The long term solution is to perform the mount namespace change and the
pivot root as this will provide greater security.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This patch implements a vhost-user-fs daemon based on Rust. It only
supports communicating through the virtqueues. The support for the
shared memory region associated with DAX will be added later.
It relies on all the code copied over from the crosvm repository, based
on the commit 961461350c0b6824e5f20655031bf6c6bf6b7c30.
It also relies on the vhost_user_backend crate, limiting the amount of
code needed to get this daemon up and running.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Add a Server type that links the FUSE protocol with the virtio
transport. It parses messages sent on the virtio queue and then
calls the appropriate method of the Filesystem trait.
This code has been ported over from crosvm commit
961461350c0b6824e5f20655031bf6c6bf6b7c30.
One small modification has been applied to the original code. Because
cloud-hypervisor didn't have the macro used by crosvm, the match
statement in the function handle_message() has been updated.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Introduce helpers to split a virtio descriptor into its readable part on
one side, and into its writable part on the other side. This is useful
to separate the request from the reply.
This code has been ported over from crosvm commit
961461350c0b6824e5f20655031bf6c6bf6b7c30.
Two important modifications have been applied to the original code:
- GuestMemory is replaced by GuestMemoryMmap from the vm-memory crate,
which comes with different ways of accessing the memory regions.
- VolatileSlice has different methods, which means the code has been
updated accordingly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The vm_memory implementation for VolatileSlice is able to read and write
to a source or destination which implements a Read or Write trait.
Unfortunately, this is not enough for this specific use case as we need
to be able to write to a file at a specific offset, which is not
provided by the Read or Write trait.
This code has been ported over from crosvm commit
961461350c0b6824e5f20655031bf6c6bf6b7c30.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Add a "passthrough" file system implementation that just forwards its
requests to the appropriate system call.
This code has been ported over from crosvm commit
961461350c0b6824e5f20655031bf6c6bf6b7c30.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Add the `Filesystem` trait, which is the main interface between the
transport and the actual file system implementation.
This code has been ported over from crosvm commit
961461350c0b6824e5f20655031bf6c6bf6b7c30.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The multikey module provides a BTreeMap implementation that can use one
of 2 different kinds of keys to look up a value. This is needed by the
virtio-fs server since it needs to be able to look up keys either by
u64 or by a (ino_t, dev_t) pair.
This code has been ported over from crosvm commit
961461350c0b6824e5f20655031bf6c6bf6b7c30.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
To be able to deal with FUSE requests, this commit introduces FUSE
definitions, copied over from the crosvm commit
961461350c0b6824e5f20655031bf6c6bf6b7c30.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This new crate will be dedicated to vhost_user_fs specific code that can
be used as a library from the vhost-user-fs daemon.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to iterate over a chain of descriptor chains, this code has
been ported over from crosvm, based on the commit
961461350c0b6824e5f20655031bf6c6bf6b7c30.
The main modification compared to the original code is the way the
sorting between readable and writable descriptors happens.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By giving the same caps to both cloud-hypervisor and the test binary, we
can access information under /proc related to the VM PID.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Instead of using bash and awk, using Rust allows us to retrieve
information about a VM process with the right permissions as we are not
forced to spawn a new child process.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The test validates that when the mergeable option is enabled, the
resulting PSS for two instances of cloud-hypervisor is lower than two
instances not using the mergeable flag.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In case the VM is started with the flag "--pmem mergeable=on", it means
the user expects the guest persistent memory pages to be marked as
mergeable. This commit relies on the madvise(MADV_MERGEABLE) system call
to inform the host kernel about these pages.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to let the user indicate if the persistent memory pages should
be marked as mergeable or not, a new option is being introduced.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In case the VM is started with the flag "--memory mergeable=on", it
means the user expects the guest RAM pages to be marked as mergeable.
This commit relies on the madvise(MADV_MERGEABLE) system call to inform
the host kernel about these pages.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to let the user indicate if the guest RAM pages should be
marked as mergeable or not, a new option is being introduced.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
When vmm.ping give a response, we expect get the version from
the VMM not the vmm create
Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
vmm.ping will help to check if http API server is up and
running.
This also removes the vmm.info endpoint.
Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
The CPU manager uses an I/O port and to prevent potential clashes with
assignment for PCI devices ensure that it is allocated by the allocator.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Rather than hardcode the CPU status for all the CPUs instead query from
the CPU manager via the I/O port that is is on via the ACPI tables.
Each CPU device has a _STA method that calls into the CSTA method which
reads and writes the I/O ports via the PRST field which exposes the I/O
port through and OpRegion.
As we only support boot CPUS report that all the CPUs are enabled for
now.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The Linux kernel expects all CPUs, whether they be enabled or disabled
to have an _MAT entry containing the LAPIC details for this CPU with the
enabled bit set to 1 (in the flags.)
In the MADT table the same bit is used to determine if the CPU is
present at boot vs available later.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This is necessary as adding support for NamedFields requires a PkgLength
calculation that does not include the length itself.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
We disabled msi/msix twice inside Drop trait for VfioPciDevice,
which resulted in error message "Could not disable MSI-X". Eliminating
this error by check whether the msi/msix capability is enabled.
Signed-off-by: Wu Zongyong <wuzongyong@linux.alibaba.com>
CC: Liu Jiang <gerry@linux.alibaba.com>
The comments of vfio kernel module said that individual subindex
interrupts can be disabled using the -1 value for DATA_EVENTFD or
the index can be disabled as a whole with:
flags = (DATA_NONE|ACTION_TRIGGER), count = 0.
Signed-off-by: Wu Zongyong <wuzongyong@linux.alibaba.com>
CC: Liu Jiang <gerry@linux.alibaba.com>
Update all references to the new repository locations. Many of these will
redirect however the one used for the hypervisor-fw binary does not so
this is required to allow the builds to pass.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This commit introduces a dedicated document describing the device model
supported by cloud-hypervisor VMM.
It needs to be updated anytime a new device will be added in the future.
Fixes#437
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
When adding devices to the guest, and populating the device model, we
should prefix the routines with add_. When we're just creating the
device objects but not yet adding them we use make_.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
In order to reduce the DeviceManager's new() complexity, we can move the
MMIO devices creation code into its own routine.
Fixes: #441
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
In order to reduce the DeviceManager's new() complexity, we can move the
PCI devices creation code into its own routine.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
In order to reduce the DeviceManager's new() complexity, we can move the
ACPI device creation code into its own routine.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
In order to reduce the DeviceManager's new() complexity, we can move the
ACPI device creation code into its own routine.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
In order to reduce the DeviceManager's new() complexity, we can move the
legacy devices creation code into its own routine.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
In order to reduce the DeviceManager's new() complexity, we can move the
console creation code into its own routine.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The cloud-hypervisor supports the DAX shared region for better virtio-fs
performances, but it was not updated in the documentation. This commit
takes care of this and also update the mount command since it changed
with virtio-fs v0.3.
Fixes#433
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Move CpuManager, Vcpu and related functionality to its own module (and
file) inside the VMM crate
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In most cases we return a 204 (No Content) and not a 201.
In those cases, we do not send any HTTP body back at all.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The new micro_http package provides a built-in HttpServer wrapper for
running a more robust HTTP server based on the package HTTP API.
Switching to this implementation allows us to, among other things,
handle HTTP requests that are larger than 1024 bytes.
Fixes: #423
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Pull details of vCPU management (booting, pausing, resuming, shutdown)
into it's own structure. This will ultimately enable this to be moved to
its own file and encapsulate all the vCPU handling for the VMM.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Remove ACPI table creation from arch crate to the vmm crate simplifying
arch::configure_system()
GuestAddress(0) is used to mean no RSDP table rather than adding
complexity with a conditional argument or an Option type as it will
evaluate to a zero value which would be the default anyway.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Use the new vhost-user-blk backend for the integration tests,
eliminating the need for building vubd using the implementation in
QEMU.
Signed-off-by: Sergio Lopez <slp@redhat.com>
set_features() fails with InvalidOperation if !self.owned. I don't see
this as a requirement in the specification and, in fact, vm-virtio
implementation for resetting the device calls SET_FEATURES just after
RESET_OWNER.
Signed-off-by: Sergio Lopez <slp@redhat.com>
Create a vhost-user-blk backend using vhost-user-backend and following
the conventions established by the existing vhost-user-net
implementation.
This backend is based on https://github.com/slp/vhost-user-backend,
but a bit simplified, making it closer to the original implementation
in Firecracker. The main features missing are EVENT_IDX, support for
asynchronous I/O and multiqueue, but it's still fully functional and
provides a good starting point for evolving it into a more complete
implementation.
Signed-off-by: Sergio Lopez <slp@redhat.com>
Extend VhostUserBackend trait with protocol_features(), so device
backend implementations can freely define which protocol features they
want to support.
Signed-off-by: Sergio Lopez <slp@redhat.com>
The VhostUserConfig carries a message with a payload, the contents of
which depend on the kind of device being emulated.
With this change, we calculate the offset of the payload within the
message, check its size corresponds to the expected one, and pass it
to the backend as a reference to a slice adjusted to the payload
dimensions.
The backend will be responsible of validating the payload, as it's the
one aware of its expected contents.
Signed-off-by: Sergio Lopez <slp@redhat.com>
Add basic processor details to the DSDT table. The code has to be
slightly convoluted (with the second pass over the cpu_devices vector)
in order to keep the objects alive long enough in order to be able to
take their reference.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In vm_reboot, while build the new vm, the old one pointed by self.vm
is not released, that is, the tap opened by self.vm is not closed
either. As a result, the associated dev name slot in host kernel is
still in use state, which prevents the new build from picking it up as
the new opened tap's name, but to use the name in next slot finally.
Call self.vm_shutdown instead here since it has call take() on vm reference,
which could ensure the old vm is destructed before the new vm build.
Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
Ensure that we tell the allocator about all the I/O ports that we are
using for I/O bus attached devices (serial, i8042, ACPI device.)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In order to group together some functions that can be shared across
virtio transport layers, this commit introduces a new trait called
VirtioTransport.
The first function of this trait being ioeventfds() as it is needed from
both virtio-mmio and virtio-pci devices, represented by MmioDevice and
VirtioPciDevice structures respectively.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that kvm-ioctls has been updated, the function unregister_ioevent()
can be used to remove eventfd previously associated with some specific
PIO or MMIO guest address. Particularly, it is useful for the PCI BAR
reprogramming case, as we want to ensure the eventfd will only get
triggered by the new BAR address, and not the old one.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We need to rely on the latest kvm-ioctls version to benefit from the
recent addition of unregister_ioevent(), allowing us to detach a
previously registered eventfd to a PIO or MMIO guest address.
Because of this update, we had to modify the current constraint we had
on the vmm-sys-util crate, using ">= 0.1.1" instead of being strictly
tied to "0.2.0".
Once the dependency conflict resolved, this commit took care of fixing
build issues caused by recent modification of kvm-ioctls relying on
EventFd reference instead of RawFd.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The specific part of PCI BAR reprogramming that happens for a virtio PCI
device is the update of the ioeventfds addresses KVM should listen to.
This should not be triggered for every BAR reprogramming associated with
the virtio device since a virtio PCI device might have multiple BARs.
The update of the ioeventfds addresses should only happen when the BAR
related to those addresses is being moved.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The PciDevice trait is supposed to describe only functions related to
PCI. The specific method ioeventfds() has nothing to do with PCI, but
instead would be more specific to virtio transport devices.
This commit removes the ioeventfds() method from the PciDevice trait,
adding some convenient helper as_any() to retrieve the Any trait from
the structure behing the PciDevice trait. This is the only way to keep
calling into ioeventfds() function from VirtioPciDevice, so that we can
still properly reprogram the PCI BAR.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Storing a strong reference to the AddressManager behind the
DeviceRelocation trait results in a cyclic reference count.
Use a weak reference to break that dependency.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the value being written to the BAR, the implementation can
now detect if the BAR is being moved to another address. If that is the
case, it invokes move_bar() function from the DeviceRelocation trait.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to trigger the PCI BAR reprogramming from PciConfigIo and
PciConfigMmmio, we need the PciBus to have a hold onto the trait
implementation of DeviceRelocation.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By implementing the DeviceRelocation trait for the AddressManager
structure, we now have a way to let the PCI BAR reprogramming happen.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the type of BAR, we can now provide the correct address related
to a BAR index provided by the caller.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to reuse the SystemAllocator later at runtime, it is moved into
the new structure AddressManager. The goal is to have a hold onto the
SystemAllocator and both IO and MMIO buses so that we can use them
later.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to support PCI BAR reprogramming, the PciDevice trait defines a
new method move_bar() dedicated for taking appropriate action when a BAR
is moved to a different guest address.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This new trait will allow the VMM to implement the mechanism following a
BAR modification regarding its location in the guest address space.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
After some of the build changes there was no longer a release artifact
left to be used for the release builds.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
To avoid a clash with to_bytes() for the unsigned integer types that is
coming in a future release.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This was verified by comparing the ASL from disassembling the DSDT
before and after. All the individual AML components themselves are also
unit tested.
Fixes: #352
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The generic version does not work in this case as it the size of the the
&[u8] is not the size of the slice's contents but how much memory the
slice object itself takes up.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add support for Word/DWord/QWord address spaces for I/O, memory and
buses. Using sensible defaults for infrequently set flags.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add support for generating resource templates (a kind of buffer) along
with generating Memory32Fixed to go into it.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Packages are a way of grouping values together and as such they require
an explicit length which is a variable length encoding calculated with
create_pkg_length().
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add support for the numerical types. By using type aliases the
generation can be very ergonomic as From<T> is implemented.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Root ("\"), single, dual and multi-part names are supported. "^" is not
supported (not widely used.)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
cargo audit audits Cargo.lock for crates with security vulnerabvilities
reported by the RustSec Advisory Database.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
A new ClearLinux image has been uploaded to the Azure storage account.
It is based off of the ClearLinux cloudguest image 31310 version, with
two extra bundles added to it.
First bundle is sysadmin-basic to include utility like netcat, and the
second bundle is iperf, adding the iperf binary to the image.
The image is 2G in size.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on recent update of the cloudguest image used by Cloud Hypervisor,
we identified some missing or incorrect details in the instructions.
This patch is here to fix those.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This patch extends the existing virtual IOMMU documentation, explaining
how the use of huge pages can drastically improve the VM boot time.
Particularly, how in case of nested VFIO, the impact is significant and
the rationales behind it.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to speed up the boot time and reduce the amount of mappings,
this patch exposes the virtio-iommu device as supporting both 2M and 4k
page sizes.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In case a VFIO devices is being attached behind a virtual IOMMU, we
should not automatically map the entire guest memory for the specific
device.
A VFIO device attached to the virtual IOMMU will be driven with IOVAs,
hence we should simply wait for the requests coming from the virtual
IOMMU.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
When VFIO devices are created and if the device is attached to the
virtual IOMMU, the ExternalDmaMapping trait implementation is created
and associated with the device. The idea is to build a hash map of
device IDs with their associated trait implementation.
This hash map is provided to the virtual IOMMU device so that it knows
how to properly trigger external mappings associated with VFIO devices.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This patch relies on the trait implementation provided for each device
which requires some sort of external update based on a map or unmap.
Whenever a MAP or UNMAP request comes through the virtqueues, it
triggers a call to the external mapping trait with map()/unmap()
functions being invoked.
Those external mappings are meant to be used from VFIO and vhost-user
devices as they need to update their own mappings. In case of VFIO, the
goal is to update the DMAR table in the physical IOMMU, while vhost-user
devices needs to update their internal representation of the virtqueues.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
With this implementation of the trait ExternalDmaMapping, we now have
the tool to provide to the virtual IOMMU to trigger the map/unmap on
behalf of the guest.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The VFIO container is the object needed to update the VFIO mapping
associated with a VFIO device. This patch allows the device manager
to have access to the VFIO container.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The new crate vm-device is created here to host the definitions of
traits not meant to be tied to virtio of VFIO specifically. We need to
add a new trait to update external DMA mappings for devices, which is
why the vm-device crate is the right fit for this.
We can expect this crate to be extended later once the design gets
approved from a rust-vmm perspective.
In this specific use case, we can have some devices like VFIO or
vhost-user ones requiring to be notified about mapping updates. This
new trait ExternalDmaMapping will allow such devices to implement their
own way to handle such event.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This patch attaches VFIO devices to the virtual IOMMU if they are
identified as they should be, based on the option "iommu=on". This
simply takes care of adding the PCI device ID to the ACPI IORT table.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Having the virtual IOMMU created with --iommu is one thing, but we also
need a way to decide if a VFIO device should be attached to the virtual
IOMMU or not. That's why we introduce an extra option "iommu" with the
value "on" or "off". By default, the device is not attached, which means
"iommu=off".
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Clean up the error handling and ensure that where possible errors are
propagated. Make use of std::convert::From in order to translate error
types.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Simplify the check for the unusual situation where the memory is not
configured by using .ok_or() on the option to convert it to a result.
This cleans up a bunch of extra indentation.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Remove messages that are left over from the development of the project
that represent normal operation for the backend. This cleans up the
console output and improves performance.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Fix invalid type for version:
- VmInfo.version.type string
Change Null value from enum as it has problems to build clients with
openapi tools.
- ConsoleConfig.mode.enum Null -> Nil
Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
We should return an explicit error when the transition from on VM state
to another is invalid.
The valid_transition() routine for the VmState enum essentially
describes the VM state machine.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
We pause a VM from the API, then SSH'ing into it should fail.
After resuming, SSH'ing should work again.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
In order to pause a VM, we signal all the vCPU threads to get them out
of vmx non-root. Once out, the vCPU thread will check for a an atomic
pause boolean. If it's set to true, then the thread will park until
being resumed.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Because the L2 VM running in the VFIO integration test is actually
running as L3 (since the CI runs in a VM), it can take quite some
time for this VM to boot.
The way to solve this issue is to extend the sleep time before to try
communicating with the L2 VM, but also to speed up the boot time by
using virtio-console instead of serial. We suspect the use of serial,
implying PIO VM exits for each character on the serial port is quite
expensive compared to the paravirtualized console.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Azure virtual machines can have private IPs in the 172.16.x.x range,
causing some issues with the VFIO test. By using 172.17.x.x for this
test, we avoid IP conflicts.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that our custom kernel includes all the patches for the full support
of virtio-iommu, we can go one step further by attaching the virtio-net
device to the virtual IOMMU and use it to SSH some commands validating
both disks and the network card are isolated into their own IOMMU group.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that cloud-hypervisor can expose a virtual IOMMU to its guest VM,
the integration test validating the VFIO support with virtio-net can be
updated to use cloud-hypervisor exclusively.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Because we want both early support for virtio-fs and virtio-iommu, our
custom kernel is now based on the kernel branch virtio-fs-virtio-iommu.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to support nested virtualization and nested device passthrough
from our CI tests, we need some extra kernel configuration options to be
enabled.
CONFIG_KVM and CONFIG_VIRTUALIZATION for nested virtualization.
CONFIG_VFIO for nested device passthrough.
CONFIG_VIRTIO_IOMMU and CONFIG_ACPI_IORT for virtio-iommu support.
With all these new options applied, we can leverage virtio-iommu to
attach some VFIO devices to it and pass them through a second layer of
virtualization.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
So that we don't need to forward an ExitBehaviour up to the VMM thread.
This simplifies the control loop and the VMM thread even further.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The firmware has been recently updated to find the EFI partition, no
matter if the disk is not the first one of the list.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This patch introduces a specific documentation for the virtual IOMMU
device. This is important to understand what the use cases are for this
new device and how to properly use it with virtio devices.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit is the glue between the virtio-pci devices attached to the
vIOMMU, and the IORT ACPI table exposing them to the guest as sitting
behind this vIOMMU.
An important thing is the trait implementation provided to the virtio
vrings for each device attached to the vIOMMU, as they need to perform
proper address translation before they can access the buffers.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Having the virtual IOMMU created with --iommu is one thing, but we also
need a way to decide if a virtio-vsock device should be attached to this
virtual IOMMU or not. That's why we introduce an extra option "iommu"
with the value "on" or "off". By default, the device is not attached,
which means "iommu=off".
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Having the virtual IOMMU created with --iommu is one thing, but we also
need a way to decide if a virtio-console device should be attached to
this virtual IOMMU or not. That's why we introduce an extra option
"iommu" with the value "on" or "off". By default, the device is not
attached, which means "iommu=off".
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Having the virtual IOMMU created with --iommu is one thing, but we also
need a way to decide if a virtio-pmem device should be attached to this
virtual IOMMU or not. That's why we introduce an extra option "iommu"
with the value "on" or "off". By default, the device is not attached,
which means "iommu=off".
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Having the virtual IOMMU created with --iommu is one thing, but we also
need a way to decide if a virtio-rng device should be attached to this
virtual IOMMU or not. That's why we introduce an extra option "iommu"
with the value "on" or "off". By default, the device is not attached,
which means "iommu=off".
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Having the virtual IOMMU created with --iommu is one thing, but we also
need a way to decide if a virtio-net device should be attached to this
virtual IOMMU or not. That's why we introduce an extra option "iommu"
with the value "on" or "off". By default, the device is not attached,
which means "iommu=off".
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Having the virtual IOMMU created with --iommu is one thing, but we also
need a way to decide if a virtio-blk device should be attached to this
virtual IOMMU or not. That's why we introduce an extra option "iommu"
with the value "on" or "off". By default, the device is not attached,
which means "iommu=off".
One side effect of this new option is that we had to introduce a new
option for the disk path, simply called "path=".
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Adding a simple iommu boolean field to the VmConfig structure so that we
can later use it to create a virtio-iommu device for the current VM.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The virtual IOMMU exposed through virtio-iommu device has a dependency
on ACPI. It needs to expose the device ID of the virtio-iommu device,
and all the other devices attached to this virtual IOMMU. The IDs are
expressed from a PCI bus perspective, based on segment, bus, device and
function.
The guest relies on the topology description provided by the IORT table
to attach devices to the virtio-iommu device.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This patch introduces the first implementation of the virtio-iommu
device. This device emulates an IOMMU for the guest, which allows
special use cases like nesting passed through devices, or even using
IOVAs from the guest.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In case some virtio devices are attached to the virtual IOMMU, their
vring addresses need to be translated from IOVA into GPA. Otherwise it
makes no sense to try to access them, and they would cause out of range
errors.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Adding virtio feature VIRTIO_F_IOMMU_PLATFORM when explicitly asked by
the user. The need for this feature is to be able to attach the virtio
device to a virtual IOMMU.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Adding virtio feature VIRTIO_F_IOMMU_PLATFORM when explicitly asked by
the user. The need for this feature is to be able to attach the virtio
device to a virtual IOMMU.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Adding virtio feature VIRTIO_F_IOMMU_PLATFORM when explicitly asked by
the user. The need for this feature is to be able to attach the virtio
device to a virtual IOMMU.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Adding virtio feature VIRTIO_F_IOMMU_PLATFORM when explicitly asked by
the user. The need for this feature is to be able to attach the virtio
device to a virtual IOMMU.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Adding virtio feature VIRTIO_F_IOMMU_PLATFORM when explicitly asked by
the user. The need for this feature is to be able to attach the virtio
device to a virtual IOMMU.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Adding virtio feature VIRTIO_F_IOMMU_PLATFORM when explicitly asked by
the user. The need for this feature is to be able to attach the virtio
device to a virtual IOMMU.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The virtio specification defines a device can be reset, which was not
supported by this vhost-user-fs implementation. The reason it is needed
is to support unbinding this device from the guest driver, and rebind it
to vfio-pci driver.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The virtio specification defines a device can be reset, which was not
supported by this vhost-user-net implementation. The reason it is needed
is to support unbinding this device from the guest driver, and rebind it
to vfio-pci driver.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The virtio specification defines a device can be reset, which was not
supported by this virtio-console implementation. The reason it is needed
is to support unbinding this device from the guest driver, and rebind it
to vfio-pci driver.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The virtio specification defines a device can be reset, which was not
supported by this virtio-vsock implementation. The reason it is needed
is to support unbinding this device from the guest driver, and rebind
it to vfio-pci driver.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The virtio specification defines a device can be reset, which was not
supported by this virtio-pmem implementation. The reason it is needed
is to support unbinding this device from the guest driver, and rebind
it to vfio-pci driver.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The virtio specification defines a device can be reset, which was not
supported by this virtio-rng implementation. The reason it is needed
is to support unbinding this device from the guest driver, and rebind
it to vfio-pci driver.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The virtio specification defines a device can be reset, which was not
supported by this virtio-net implementation. The reason it is needed is
to support unbinding this device from the guest driver, and rebind it to
vfio-pci driver.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
As of commit 2b94334a, Firecracker includes all the changes we need.
We can now switch to using it instead of carrying a copy.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
We used to have errors definitions spread across vmm, vm, api,
and http.
We now have a cleaner separation: All API routines only return an
ApiResult. All VM operations, including the VMM wrappers, return a
VmResult. This makes it easier to carry errors up to the HTTP caller.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
In order to support further use cases where a VM configuration could be
modified through the HTTP API, we only store the passed VM config when
being asked to create a VM. The actual creation will happen when booting
a new config for the first time.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Use a more idiomatic "let Ok(foo) = result" construct for:
105 | if try_numeric.is_ok() {
| ------------------- the check is happening here
106 | self.content_length = try_numeric.unwrap();
| ^^^^^^^^^^^^^^^^^^^^
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
We use the serde crate to serialize and deserialize the VmVConfig
structure. This structure will be passed from the HTTP API caller as a
JSON payload and we need to deserialize it into a VmConfig.
For a convenient use of the HTTP API, we also provide Default traits
implementations for some of the VmConfig fields (vCPUs, memory, etc...).
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The linux_loader crate Cmdline struct is not serializable.
Instead of forcing the upstream create to carry a serde dependency, we
simply use a String for the passed command line and build the actual
CmdLine when we need it (in vm::new()).
Also, the cmdline offset is not a configuration knob, so we remove it.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
They point to a vm_virtio structure (VhostUserConfig) and in order to
make the whole config serializable (through the serde crate for
example), we'd have to add a serde dependency to the vm_virtio crate.
Instead we use a local, serializable structure and convert it to
VhostUserConfig from the DeviceManager code.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The kernel path was the only mandatory command line option.
With the addition of the --api-socket option, we can run without a
kernel path and get it later through the API.
Since we can end up with VM configurations that are no longer valid by
default, we need to provide a validation check for it. For now, if the
kernel path is not defined, the VM configuration is invalid.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The API server will unconditionally run through a UNIX domain socket
which default path is /run/user/<uid>/cloud-hypervisor.<pid>.
The --api-socket command line option allows to override that default
value with some custom socket path.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
With the API server socket option, we will be able to support a model
where the user can start cloud-hypervisor with no options or an
alternative API server socket path. In this case, we don't want to try
to start a new guest VM, and for that we need to know if the user has
set any VM configuration at all. Grouping all VM configuration specific
options together is one way to be able to know about it.
If the user has not set any VM configuration, we only start the API
server. If it has set anything, we will verify that the overall
configuration is valid and will implicitly convert that configuration
into a request to the API server.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The Cloud Hyper HTTP server runs a synchronous, multi-threaded
loop that receives HTTP requests and tries to call the corresponding
endpoint handlers for the requests URIs.
An endpoint handler will parse the HTTP request and potentially
translate it into and IPC request. The handler holds an notifier and an
mspc Sender for respectively notifying and sending the IPC payload to
the VMM API server. The handler then waits for an API server response
and translate it back into an HTTP response.
The HTTP server is responsible for sending the reponse back to the
caller.
The HTTP server uses a static routes hash table that maps URIs to
endpoint handlers.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Based on Firecracker commit 58edf03b.
We're going to use the micro_http crate to serve the cloud-hypervisor
HTTP API.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The cloud-hypervisor API uses HTTP as a transport and is accessible
through a local UNIX socket.
The API root path is /api/v1 and is a collection of RPC-style methods.
All methods are static, unlike typical REST APIs. Variable (e.g. device
IDs) are passed through the request body.
Fixes: #244
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Based off of crosvm revision b5237bbcf074eb30cf368a138c0835081e747d71
add a CMOS device. This environments that can't use KVM clock to get the
current time (e.g. Windows and EFI.)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
We no longer build vubridge, so we end up cloning qemu and building
virtiofs and the block backend all the time.
Fixes: #312
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
An integration test relying on the new vhost-user-net backend now
replaces the previous test using the QEMU test backend. This allows
us to avoid building the QEMU backend, and we now really exercise the
vhost-user-net implementation as it is used for the ssh communication
in this test.
Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Create vhost-user-net backend with Tap interface, to offload network
transaction from cloud-hypervisor. The goal is to provide flexibility
about the backend being in use, but also more security as it will allow
users to isolate the backend with different security profiles since it
will run as a dedicated process on the host.
Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
While implement vhost-user-net backend with Tap interface, it keeps
failed to enable the tx vring, since there is a checking in
slave_req_handler.rs to require acked_protocol_features to be setup
as a pre-requirement, which is filled by set_protocol_features call.
Add this call in vhost-user-net device implementation to address the issue.
Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
Remove workspace from vhost_user_backend/Cargo.toml to have
vhost-user-backend compiled in cloud-hypervisor. Add workspace in
Cargo.toml to have vhost-user-backend consumed by vhost-user-net.
Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
The PCI Express Firmware specification says that the region may
be included in the E820 tables (but it must always be in the ACPI
tables.)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The PCI Express Firmware spec says that the region to be used for PCI
MMCONFIG should be reserved as part of the motherboard's resources in
the ACPI tables.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Refactor the PCI datastructures to move the device ownership to a PciBus
struct. This PciBus struct can then be used by both a PciConfigIo and
PciConfigMmio in order to expose the configuration space via both IO
port and also via MMIO for PCI MMCONFIG.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The PCI MMCONFIG area must be below 4GiB and must not be part of the
device space. Shrink the device area and put the PCI MMCONFIG region
above it.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Patch the table with the currently used constants. This will be relevant
when we want to adjust the size of the PCI device area to accomodate the
PCI MMCONFIG region.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This commit fixes all the remaining issues that were found as part of
the integration with vhost-user-net.
It fixes the way to notify that a vring is used, by using the proper
EventFd.
It removes the process_queue() function from the trait, since the
complexity it was introducing was leading to deadlocks with mutexes.
It moves the register/unregister functions for registering custom events
from the backend, from the VringEpollHandler to the VringWorker. This
allows for a lot of simplification and solve a deadlock issue.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The original logic does not has any problem without offset, since the
current offset is zero. However, if offset is not zero, while convert
vmm address to backend process address, it needs to consider the
offset.
Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
The error handling here to trigger break epoll seems not correct,
epoll will be ended once one event is handled, no matter successfully
or failed. Fix it.
Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
The vhost-user protocol does not indicate set_features could not
be issued more than once, the checking is not needed at all, and
prevent communication between master and slave. Remove it to
fix the issue.
Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
This patch modifies the library so that a consumer can update the
backend after it's been passed to the daemon.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By letting the consumer of this crate getting access to the vring
handler, we will be able to let it perform several actions without
producing a deadlock.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We cannot expect every backend to support GET_CONFIG and SET_CONFIG
commands. That's why this patch adds some default implementations for
the trait VhostUserBackend regarding both get_config() and set_config()
functions.
Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The code needs to initialize a listener to accept connection from the
VMM being the client in this case.
Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Every time an event is triggered, it needs to be read, but only based on
the status of the vring (enabled or not) will decide if the queue needs
to be processed.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The Queue structure already contains a field "ready" that can be used to
track the status of the vrings.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Let's be realistic, the trait VhostUserBackend will need to have mutable
self for some functions like handle_event, process_queue and set_config,
which is the reason why this commit needs to introduce a RwLock on the
backend instance that was passed around as a simple Arc.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Instead of locking every queues whenever something needs to be updated,
this patch modifies the code design to lock each Vring independently.
This allows for much finer granularity, and will allow multiple queues
to be handled at the same time.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The purpose of this new crate is to provide a common library to all
vhost-user backend implementations. The more is handled by this library,
the less duplication will need to happen in each vhost-user daemon.
This crate relies a lot on vhost_rs, vm-memory and vm-virtio crates.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to avoid introducing a dependency on arch in the devices crate
pass the constant in to the IOAPIC device creation.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
These are part of RAM and are used as the initial page table entries for
booting the OS and firmware (identity mapping.)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Using the existing layout module start documenting the major regions of
RAM and those areas that are reserved. Some of the constants have also
been renamed to be more consistent and some functions that returned
constant variables have been replaced.
Future commits will move more constants into this file to make it the
canonical source of information about the memory layout.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Updated clippy does not like the declaration of a "to_string()" function
and instead requires fmt::Display to be implemented.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
clippy wants us to use "if let Some(fd) = fd {}" but to combine that
with && is an experimental feature.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
We now start the main VMM thread, which will be listening for VM and IPC
related events.
In order to start the configured VM, we no longer directly call the VM
API but we use the IPC instead, to first create and then start a VM.
Fixes: #303
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Based on the newly defined Cloud Hypervisor IPC, those helpers send
VmCreate and VmStart requests respectively. This will be used by the
main thread to create and start a VM based on the CLI parameters.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
This starts the main, single VMM thread, which:
1. Creates the VMM instance
2. Starts the VMM control loop
3. Manages the VMM control loop exits for handling resets and shutdowns.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Unlike the Vmm structure we removed with commit bdfd1a3f, this new one
is really meant to represent the VM monitoring/management object.
For that, we implement a control loop that will replace the one that's
currently embedded within the Vm structure itself.
This will allow us to decouple the VM lifecycle management from the VM
object itself, by having a constantly running VMM control loop.
Besides the VM specific events (exit, reset, stdin for now), the VMM
control loop also handles all the Cloud Hypervisor IPC requests.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The VMM thread and control loop will be the sole consumer of the
EpollContext and EpollDispatch API, so let's move it to lib.rs.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Cloud Hypervisor IPC is a simple, mpsc based protocol for threads to
send command to the furture VMM thread. This patch adds the API
definition for that IPC, which will be used by both the main thread
to e.g. start a new VM based on the CLI arguments and the future HTTP
server to relay external requests received from a local Unix domain
socket.
We are moving it to its own "api" module because this is where the
external API (HTTP based) will also be implemented.
The VMM thread will be listening for IPC requests from an mpsc receiver,
process them and send a response back through another mpsc channel.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
As we're going to move the control loop to the VMM thread, the exit and
reset EventFds are no longer going to be owned by the VM.
We pass a copy of them when creating the Vm instead.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
In order to handle the VM STDIN stream from a separate VMM thread
without having to export the DeviceManager, we simply add a console
handling method to the Vm structure.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
In order to transfer the control loop to a separate VMM thread, we want
to shrink the VM control loop to a bare minimum.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Once passed to the VM creation routine, a VmConfig structure is
immutable. We can simply carry a Arc of it instead of a reference.
This also allows us to remove any lifetime bound from our VM.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The Vmm structure is just a placeholder for the KVM instance. We can
create it directly from the VM creation routine instead.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
We can integrate the kernel loading into the VM start method.
The VM start flow is then: Vm::new() -> vm.start(), which feels more
natural.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Convert Path to PathBuf and remove the associated lifetime.
Now we can remove the VmConfig associated lifetime.
Fixes#298
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Now that vhost-user has been fixed regarding size of the virtqueues,
booting a VM with the firmware from a vhost-user-blk backend actually
works. That's why this commit updates the previously introduced
integration test to make it use the firmware instead of direct kernel
boot.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The vhost-user implementation was always passing the maximum size
supported by the virtqueues to the backend, but this is obviously wrong
as it must pass the size being set by the driver running in the guest.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The 32 or 64 bits type for the memory BAR was not set correctly. This
patch ensure the right type is applied to the BAR.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
If we expect the vhost-user-blk device to be used for booting a VMM
along with the firmware, then need the device to support being reset.
In the vhost-user context, this means the backend needs to be informed
the vrings are disabled and stopped, and the owner needs to be reset
too.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit extends the existing integration test related to
vhost-user-blk by validating the block image contains one file
"foo" containing "bar".
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Currently we need use backend device from Qemu to test vhost-user-blk
device. Once the rust backend is ready, we will replace it.
Signed-off-by: Yang Zhong <yang.zhong@intel.com>
In vhost-user-blk, only WCE value can be set back to device in
guest kernel like
echo "write through" > /sys/block/vda/cache_type
So write_config() will only set WCE value from guest kernel to
vhost user side.
Signed-off-by: Yang Zhong <yang.zhong@intel.com>
Since config space in vhost-user-blk are mostly from backend
device, this change will get config space info from backend
by vhost-user protocol.
Signed-off-by: Yang Zhong <yang.zhong@intel.com>
Since vhost-user-blk use same error definition with vhost-user-net,
those errors need define to common usage.
Signed-off-by: Yang Zhong <yang.zhong@intel.com>
vhost-user-blk has better performance than virtio-blk, so we need
add vhost-user-blk support with SPDK in Rust-based VMMs.
Signed-off-by: Yang Zhong <yang.zhong@intel.com>
Add to the vhost_rs crate the INFLIGHT_SHMFD protocol feature that was
missing from the list. The feature has been recently introduced in
vhost-user specification, which is the reason why it was missing.
Signed-off-by: Yang Zhong <yang.zhong@intel.com>
This reverts commit 3e99098bf3 because we
realized there was no need for those extra functions to be introduced as
the existing one from the vhost crate are sufficient for the vhost-user
configuration use case.
Now that virtio-bindings is a crate part of the rust-vmm project, we
want to rely on this one instead of the local one we had so far.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that we have expanded our address space we should add automated
testing to try VMs that use a large amount of RAM.
As the hypervisor does an anonymous mmap() for the backing of memory and
during a typical test boot the guest will not touch it all it should be
possible to test large RAM VMs even if that exceeds the RAM of the host
machine.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The last byte was missing from the E820 RAM area. This was due to the
function using the last address relative to the first address in the
range to calculate the size. This incorrectly calculated the size by
one. This produced incorrect E820 tables like this:
[ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009ffff] usable
[ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000001ffffffe] usable
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Probe for the size of the host physical address range and use that to
establish the address range for the VM. This removes the limitation on
the size of the VM RAM and gives more space for the devices.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Currently all devices and guest memory share the same 64GiB
allocation. With guest memory working upwards and devices working
downwards. This creates issues if you want to either have a VM with a
large amount of memory or want to have devices with a large allocation
(e.g. virtio-pmem.)
As it is possible for the hypervisor to place devices anywhere in its
address range it is required for simplistic users like the firmware to
set up an identity page table mapping across the full range. Currently
the hypervisor sets up an identify mapping of 1GiB which the firmware
extends to 64GiB to match the current address space size of the
hypervisor.
A simpler solution is to place the device needed for booting with the
firmware (virtio-block) inside the 32-bit memory hole. This allows the
firmware to easily access the block device and paves the way for
increasing the address space beyond the current 64GiB limit.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
After the 32-bit gap the memory is shared between the devices and the
RAM. Ensure that the ACPI tables correctly indicate where the RAM ends
and the device area starts by patching the precompiled tables. We get
the following valid output now from the PCI bus probing (8GiB guest)
[ 0.317757] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]
[ 0.319035] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]
[ 0.320215] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]
[ 0.321431] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window]
[ 0.322613] pci_bus 0000:00: resource 8 [mem 0x240000000-0xfffffffff window]
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
There was an off-by-error in the result making the hole one byte too
big and ending at an address too high.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The starting length of the MCFG table was too long resulting in the
kernel trying to get extra MCFG entries from the table that weren't
there resulting in the following error message from the kernel:
PCI: no memory for MCFG entries
The MCFG table also has an 8 bytes of padding at the start before the
table begins.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Rerrange "use" statements and make rename variables and fields to
indicate they might be unused.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This removes the register_devices() function with all that functionality
spread across the places where the devices are created.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Mark exit_evt with an underscore it may be unused (it is ignored if the
"acpi" feature is not turned on.)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Label tests that definitely can't function with virtio-mmio (because
they use the firmware) and within those that can be used mark individual
assertions that will no longer hold (around PCI.)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add (non-default) support for using MMIO for virtio devices. This can be
tested by:
cargo build --no-default-features --features "mmio"
All necessary options will be included injected into the kernel
commandline.
Fixes: #243
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Derived from the crosvm code at 5656c124af2bb956dba19e409a269ca588c685e3
and adapted to work within cloud-hypervisor:
Main differences:
* Interrupt handling is done via a VirtioInterrupt turned into a
devices::Interrupt
* GuestMemory -> GuestMemoryMmap
* Differences in read/write for BusDevice
* Different crates for EventFd and GuestAddress
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This is necesary to be able easily translate an Interrupt to a
VirtioInterrupt which is already Sync.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This is a quick guide on how to create a custom Clear Linux image based
on the official tooling provided by Clear Linux. If for any reason, the
image we are using is missing some interesting bundles that are packaged
by Clear Linux, this documentation will be the guide on how to proceed.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By relying on a decompressed image, this patch assumes that downloading
the image will always be faster than decompressing it from the worker
VM.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Rather than calling it at the very start of the VM execution (i.e. when
the VCPUs are created) do it as part of the DeviceManager creation.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Create the virtio devices independently of adding them to the PCI bus.
Instead accrue the devices in a vector and add them to the bus en-masse.
This will allow the virtio device creation to be used independently of
PCI based transport.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This patch extends the current set of integration tests to correctly
validate that virtio-vsock is functional. It establishes a communication
between host and guest relying on the newly integrated vsock device.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Rely on the newly generated Clear Linux image for the integration
testing of cloud-hypervisor. The image has been generated using the
Clear Linux clr-installer tooling, which means it is in compliance with
the Clear Linux licensing.
This new image contains one more bundle that was not part of the default
cloudguest image. This bundle is basic-sysadmin, and contains both nc
and socat utilities.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to interact between host and guest through socket connection,
socat is a useful utility needed for integration testing.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to make the tests more verbose and help identify more quickly
where a test might be failing, this patch adds the ability for the unit
tests to print useful information with println.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The goal here is to decouple the Guest instance from the ssh connection
to send some commands to the guest. The reason being to allow ssh
commands to be issued from a different thread, which can be useful to
wait for the end of a command with a thread.join().
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
As part of the reboot test check that the binary cleanly terminated
after the subsequent shutdown.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Rather than sending a signal to the signal handler used for handling
SIGWINCH calls instead use the crate provided termination method. This
also unregisters the signal handler which also means that there won't be
a leaked signal handler remaining.
This leaked signal handler is what was causing a failure to cleanup up
the thread on subsequent requests breaking two reboots in a row.
Fixes: #252
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
With ACPI disabled there is no way to support both reset and shutdown so
make the VMM exit if the VM is rebootet (via i8042 or triple-fault
reset.)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This unit testing porting effort is based off of Firecracker commit
1e1cb6f8f8003e0bdce11d265f0feb23249a03f6
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This is the last step connecting the dots between the virtio-vsock
device and the bulk of the logic hosted in the unix and csm modules.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit relies on the new vsock::unix module to create the backend
that will be used from the virtio-vsock device.
The concept of backend is interesting here as it would allow for a vhost
kernel backend to be plugged if that was needed someday.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This code porting is based off of Firecracker commit
1e1cb6f8f8003e0bdce11d265f0feb23249a03f6
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This code porting is based off of Firecracker commit
1e1cb6f8f8003e0bdce11d265f0feb23249a03f6
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
There is a lot of code related to this virtio-vsock hybrid
implementation, that's why it's better to keep it under its
own module.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on previous patch introducing the new flag "--vsock", this commit
creates a new virtio-vsock device based on the presence of this flag.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The new flag vsock is meant to be used in order to create a VM with a
virtio-vsock device attached to it. Two parameters are needed with this
device, "cid" representing the guest context ID, and "sock" representing
the UNIX socket path which can be accessed from the host.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This is the first commit introducing the support for virtio-vsock.
This is based off of Firecracker commit
1e1cb6f8f8003e0bdce11d265f0feb23249a03f6
Fixes#102
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The default number of MSI-X vector allocated was 2, which is the minimum
defined by the virtio specification. The reason for this minimum is that
virtio needs at least one interrupt to signal that configuration changed
and at least one to specify something happened regarding the virtqueues.
But this current implementation is not optimal because our VMM supports
as many MSI-X vectors as allowed by the MSI-X specification (2048 max).
For that reason, the current patch relies on the number of virtqueues
needed by the virtio device to determine the right amount of MSI-X
vectors needed. It's important not to forget the dedicated vector for
any configuration change too.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Refactor out DeviceManager into it's own file. This is part of a bigger
effort to reduce complexity in the vm.rs file but will also allow future
separation to allow making PCI support optional.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The command "cargo build --no-default-features" does not recursively
disable the default features across the workspace. Instead add an acpi
feature at the top-level, making it default, and then make that feature
conditional on all the crate acpi features.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
2019-09-05 08:58:47 -07:00
294 changed files with 75775 additions and 27165 deletions
- [Firecracker and crosvm](#firecracker-and-crosvm)
- [5. Community](#5-community)
- [Contribute](#contribute)
- [Join us](#join-us)
# 1. What is Cloud Hypervisor?
**This project is an experiment and should not be used with production workloads.**
Cloud Hypervisor is an open source Virtual Machine Monitor (VMM) that runs on top of [KVM](https://www.kernel.org/doc/Documentation/virtual/kvm/api.txt).
The project focuses on exclusively running modern, cloud workloads, on top of a limited set of hardware architectures and platforms.
Cloud workloads refers to those that are usually run by customers inside a cloud provider. For our purposes this means modern
Linux* distributions with most I/O handled by paravirtualised devices (i.e. virtio), no requirement for legacy devices and recent CPUs and KVM.
Cloud workloads refers to those that are usually run by customers inside a cloud provider. For our purposes this means modern operating systems with most I/O handled by paravirtualised devices (i.e. virtio), no requirement for legacy devices, and 64-bit CPUs.
Cloud Hypervisor is implemented in [Rust](https://www.rust-lang.org/) and is based on the [rust-vmm](https://github.com/rust-vmm) crates.
@@ -36,27 +36,24 @@ Cloud Hypervisor is implemented in [Rust](https://www.rust-lang.org/) and is bas
### High Level
* KVM and KVM only based
* Minimal emulation
* Low latency
* Low memory footprint
* Low complexity
* High performance
* Small attack surface
* 64-bit support only
* Build time configurable CPU, memory, PCI and NVDIMM hotplug
* Machine to machine migration
- KVM based
- Minimal emulation
- Low latency
- Low memory footprint
- Low complexity
- High performance
- Small attack surface
- 64-bit support only
- CPU, memory, PCI hotplug
- Machine to machine migration
### Architectures
`cloud-hypervisor` only supports the `x86-64`CPU architecture for now.
We're planning to add support for the `AArch64` architecture in the future.
Cloud Hypervisor supports the `x86-64`and `AArch64` architectures. There are some small differences in functionality between the two architectures (see [#1125](https://github.com/cloud-hypervisor/cloud-hypervisor/issues/1125)).
### Guest OS
*`64-bit Linux`
Support for *modern* 64-bit Windows guest is being evaluated.
Cloud Hypervisor supports `64-bit Linux` with support for _modern_ 64-bit Windows guests currently under development.
Multiple arguments can be given to the `--disk` parameter, currently the firmware requires that the bootable image is on the first disk.
Multiple arguments can be given to the `--disk` parameter.
### Custom kernel and disk image
#### Building your kernel
`cloud-hypervisor` also supports direct kernel boot into a `vmlinux` ELF kernel
image. You want to build such an image first:
Cloud Hypervisor also supports direct kernel boot into a `vmlinux` ELF kernel or `bzImage`. In order to support virtio-fs and virtio-iommu we have our own development branch. You are of course able to use your own kernel but these instructions will continue with the version that we develop and test against.
Direct kernel boot to userspace should work with a rootfs from most distributions.
## Hot Plug
Cloud Hypervisor supports hotplug of CPUs, passthrough devices (VFIO), `virtio-{net,block,pmem,fs,vsock}` and memory resizing. This [document](https://github.com/cloud-hypervisor/cloud-hypervisor/blob/master/docs/hotplug.md) details how to add devices to
a running VM.
## Device Model
Details of the device model can be found in this [documentation](https://github.com/cloud-hypervisor/cloud-hypervisor/blob/master/docs/device_model.md).
## TODO
We are not tracking the `cloud-hypervisor` TODO list from a specific git tracked file but through
In order to satisfy the design goal of having a high-performance, security-focused hypervisor the decision
was made to use the [Rust](https://www.rust-lang.org/) programming language.
The language's strong focus on memory and thread safety makes it an ideal candidate for implementing VMMs
The language's strong focus on memory and thread safety makes it an ideal candidate for implementing VMMs.
Instead of implementing the VMM components from scratch, `cloud-hypervisor` is importing the [rust-vmm](https://github.com/rust-vmm)
Instead of implementing the VMM components from scratch, Cloud Hypervisor is importing the [rust-vmm](https://github.com/rust-vmm)
crates, and sharing code and architecture together with other VMMs like e.g. Amazon's [Firecracker](https://firecracker-microvm.github.io/)
and Google's [crosvm](https://chromium.googlesource.com/chromiumos/platform/crosvm/).
`cloud-hypervisor` embraces the rust-vmm project goals, which is to be able to share and re-use
as many virtualization crates as possible. As such, the `cloud-hypervisor` relationship with the rust-vmm
Cloud Hypervisor embraces the rust-vmm project goals, which is to be able to share and re-use
as many virtualization crates as possible. As such, the Cloud Hypervisor relationship with the rust-vmm
project is twofold:
1. It will use as much of the rust-vmm code as possible. Any new rust-vmm crate that's relevant to the project
goals will be integrated as soon as possible.
2. As it is likely that the rust-vmm project will lack some of the features that `cloud-hypervisor` needs (e.g. ACPI,
VFIO, vhost-user, etc), we will be using the `cloud-hypervisor` VMM to implement and test them, and contribute them
2. As it is likely that the rust-vmm project will lack some of the features that Cloud Hypervisor needs (e.g. ACPI,
VFIO, vhost-user, etc), we will be using the Cloud Hypervisor VMM to implement and test them, and contribute them
back to the rust-vmm project.
## Firecracker and crosvm
A large part of the `cloud-hypervisor` code is based on either the Firecracker or the crosvm projects implementations.
A large part of the Cloud Hypervisor code is based on either the Firecracker or the crosvm projects implementations.
Both of these are VMMs written in Rust with a focus on safety and security, like Cloud Hypervisor.
However we want to emphasize that the Cloud Hypervisor project is neither a fork nor a reimplementation of any of those
@@ -246,6 +301,11 @@ crates as possible.
# 5. Community
The Cloud Hypervisor project follows the governance, and community guidelines described in
the [Community](https://github.com/cloud-hypervisor/community) repository.
## Contribute
We are working on building a global, diverse and collaborative community around the Cloud Hypervisor project.
Anyone who is interested in [contributing](CONTRIBUTING.md) to the project is welcome to participate.
@@ -255,29 +315,5 @@ etc, are all equal and welcome means of contribution. See the [CONTRIBUTING](CON
## Join us
Get an [invite to our Slack channel](https://join.slack.com/t/cloud-hypervisor/shared_invite/enQtNjY3MTE3MDkwNDQ4LTc0YzlmYzQxZDkxNDVhYzZjZjA5MTkxMGY3NTI3YzMzYTFkM2IyY2E0YTIxMzkyYTEwYzdlMzBhMWYxYzVmNDI)
Get an [invite to our Slack channel](https://join.slack.com/t/cloud-hypervisor/shared_invite/enQtNjY3MTE3MDkwNDQ4LWQ1MTA1ZDVmODkwMWQ1MTRhYzk4ZGNlN2UwNTI3ZmFlODU0OTcwOWZjMTkwZDExYWE3YjFmNzgzY2FmNDAyMjI)
and [join us on Slack](https://cloud-hypervisor.slack.com/).
# 6. Security
**Reporting a Potential Security Vulnerability**: If you have discovered
potential security vulnerability in this project, please send an e-mail to
secure@intel.com. For issues related to Intel Products, please visit
https://security-center.intel.com.
It is important to include the following details:
- The projects and versions affected
- Detailed description of the vulnerability
- Information on known exploits
Vulnerability information is extremely sensitive. Please encrypt all security
vulnerability reports using our *PGP key*
A member of the Intel Product Security Team will review your e-mail and
contact you to to collaborate on resolving the issue. For more information on
how Intel works to resolve security issues, see: *Vulnerability Handling
If you want to build and test Cloud Hypervisor without having to install all the required dependencies, you can also turn to the development script: dev_cli.sh.
To build the development container:
```bash
./scripts/dev_cli.sh build-container
```
To build Cloud-hypervisor in the container:
```bash
./scripts/dev_cli.sh build
```
## Run
Assuming you have built Cloud-hypervisor with the development container, a VM can be started with command:
@@ -10,11 +10,12 @@ __virtio-fs__, also known as __vhost-user-fs__ is a virtual device defined by th
This virtual device relies on the _vhost-user_ protocol, which assumes the backend (device emulation) is handled by a dedicated process running on the host. This daemon is called __virtiofsd__ and needs to be present on the host.
The `cache=none` option here is an important one as it tells the daemon not to try any memory mapping of the files, but instead to use the _virtqueues_ to convey the files content. The support for the memory mapping of the files will be added later.
The `cache=none` option should be the default when using `virtiofsd` with the __cloud-hypervisor__ VMM. This prevents from using the guest page cache, which reduces the memory footprint of the guest. When running multiple virtual machines on the same host, this will let the host deal with page cache, which will increase the density of virtual machines which can be launched.
The `cache=always` option will allow for the guest page cache to be used, which will increase the memory footprint of the guest. This option should be used only for specific use cases where a single VM is going to be running on a host.
### The kernel
In order to leverage __virtio-fs__ support from within the guest, and because the code has not been merged in upstream Linux kernel yet, it is required to build a custom kernel embedding the patches.
The following branch `virtio-pmem_and_virtio-fs` on the repository https://github.com/sboeuf/linux.git includes all the needed patches to support __virtio-fs__.
The following branch `virtio-fs-virtio-iommu` on the repository https://github.com/cloud-hypervisor/linux.git includes all the needed patches to support __virtio-fs__.
Make sure to build a kernel out of this branch that can be then used to boot the VM.
@@ -48,23 +52,37 @@ Direct kernel boot option is preferred since we need to provide the custom kerne
Because _vhost-user_ expects a dedicated process (__virtiofsd__ in this case) to be able to access the guest RAM to communicate through the _virtqueues_ with the driver running in the guest, `--memory` option needs to be slightly modified. It needs to specify a backing file for the memory so that an external process can access it.
Assuming you have `clear-kvm.img` and `custom-vmlinux.bin` on your system, here is the __cloud-hypervisor__ command you need to run:
Assuming you have `focal-server-cloudimg-amd64.raw` and `custom-vmlinux.bin` on your system, here is the __cloud-hypervisor__ command you need to run:
By default, DAX is enabled with a cache window of 8GiB. You can specify a custom size (let's say 4GiB for this example) for the cache by explicitly setting DAX and the cache size:
In case you don't want to use a shared window of cache to pass the shared files content, this means you will have to explicitly disable DAX with `dax=off`. Note that in this case, the `cache_size` parameter will be ignored.
The `tag` needs to be consistent with what has been provided through the __cloud-hypervisor__ command line.
The `tag` needs to be consistent with what has been provided through the __cloud-hypervisor__ command line, which happens to be `myfs` in this example.
The `-o dax` option must be removed in case the shared cache region is not enabled from the VMM.
Currently Cloud Hypervisor only support hot plugging of CPU devices.
## Kernel support
For hotplug on Cloud Hypervisor ACPI GED support is needed. This can either be achieved by turning on `CONFIG_ACPI_REDUCED_HARDWARE_ONLY`
or by using this kernel patch (available in 5.5-rc1 and later): https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/patch/drivers/acpi/Makefile?id=ac36d37e943635fc072e9d4f47e40a48fbcdb3f0
## CPU Hot Plug
Extra vCPUs can be added and removed from a running Cloud Hypervisor instance. This is controlled by two mechanisms:
1. Specifying a number of maximum potential vCPUs that is greater than the number of default (boot) vCPUs.
2. Making a HTTP API request to the VMM to ask for the additional vCPUs to be added.
To use CPU hotplug start the VM with the number of max vCPUs greater than the number of boot vCPUs, e.g.
The extra vCPU threads will be created and advertised to the running kernel. The kernel does not bring up the CPUs immediately and instead the user must "online" them from inside the VM:
```shell
root@ch-guest ~ # lscpu | grep list:
On-line CPU(s) list: 0-3
Off-line CPU(s) list: 4-7
root@ch-guest ~ # echo 1 | tee /sys/devices/system/cpu/cpu[4,5,6,7]/online
1
root@ch-guest ~ # lscpu | grep list:
On-line CPU(s) list: 0-7
```
After a reboot the added CPUs will remain.
Removing CPUs works similarly by reducing the number in the "desired_vcpus" field of the reisze API. The CPUs will be automatically offlined inside the guest so there is no need to run any commands inside the guest:
As per adding CPUs to the guest, after a reboot the VM will be running with the reduced number of vCPUs.
## Memory Hot Plug
### ACPI method
Extra memory can be added from a running Cloud Hypervisor instance. This is controlled by two mechanisms:
1. Allocating some of the guest physical address space for hotplug memory.
2. Making a HTTP API request to the VMM to ask for a new amount of RAM to be assigned to the VM. In the case of expanding the memory for the VM the new memory will be hotplugged into the running VM, if reducing the size of the memory then change will take effect after the next reboot.
To use memory hotplug start the VM specifying some size RAM in the `hotplug_size` parameter to the memory configuration. Not all the memory specified in this parameter will be available to hotplug as there are spacing and alignment requirements so it is recommended to make it larger than the hotplug RAM needed.
Because the ACPI method is the default, there is no need to add the extra option `hotplug_method=acpi`.
The new memory is now available to use inside the VM:
```shell
free -h
total used free shared buff/cache available
Mem: 3.0Gi 71Mi 2.8Gi 0.0Ki 47Mi 2.8Gi
Swap: 32Mi 0B 32Mi
```
Due to guest OS limitations is is necessary to ensure that amount of memory added (between currently assigned RAM and that which is desired) is a multiple of 128MiB.
The same API can also be used to reduce the desired RAM for a VM but the change will not be applied until the VM is rebooted.
Memory and CPU resizing can be combined together into the same HTTP API request.
### virtio-mem method
Extra memory can be added and removed from a running Cloud Hypervisor instance. This is controlled by two mechanisms:
1. Allocating some of the guest physical address space for hotplug memory.
2. Making a HTTP API request to the VMM to ask for a new amount of RAM to be assigned to the VM.
To use memory hotplug start the VM specifying some size RAM in the `hotplug_size` parameter along with `hotplug_method=virtio-mem` to the memory configuration.
The new memory is now available to use inside the VM:
```shell
free -h
total used free shared buff/cache available
Mem: 3.0Gi 71Mi 2.8Gi 0.0Ki 47Mi 2.8Gi
Swap: 32Mi 0B 32Mi
```
The same API can also be used to reduce the desired RAM for a VM. It is important to note that reducing RAM size might only partially work, as the guest might be using some of it.
## PCI Device Hot Plug
Extra PCI devices can be added and removed from a running Cloud Hypervisor instance. This is controlled by making a HTTP API request to the VMM to ask for the additional device to be added, or for the existing device to be removed.
To use PCI device hotplug start the VM with the HTTP server.
00:01.0 Unassigned class [ffff]: Red Hat, Inc. Virtio console (rev 01)
00:02.0 Mass storage controller: Red Hat, Inc. Virtio block device (rev 01)
00:03.0 Unassigned class [ffff]: Red Hat, Inc. Virtio RNG (rev 01)
```
After a reboot the added PCI device will remain.
### Remove PCI device
Removing a PCI device works the same way for all kind of PCI devices. The unique identifier related to the device must be provided. This identifier can be provided by the user when adding the new device, or by default Cloud Hypervisor will assign one.
* Developers who want to understand what log level to use and when,
* Users who want to debug issues with running their workloads in Cloud Hypervisor
## Control
The number of `-v` parameters passed to the `cloud-hypervisor` binary will determine the log level. Currenly the default is log messages up to `WARN:` (`warn!`) are included by default. The `--log-file` allows the log to be sent to a location other than `stderr`.
## Levels
### `error!()`
For immediate, unrecoverable errors where it does not make sense for the execution to continue as the behaviour of the VM is considerablely impacted.
Cloud Hypervisor should exit shortly after reporting this error (with a non-zero exit code). Generally this should be used during initial construction of the VM state before the virtual CPUs have begun running code.
A typical situation where this might occur is when the user is using command line options that conflict with each other or is trying to use a file that is not present on the filesystem.
Users should react to this error by checking their initial VM configuration.
### `warn!()`
A serious problem has occured but the execution of the VM can continue although some functionality might be impacted.
A typical example of where this level of message should be generated is during an API call request that cannot be fulfilled.
The user should investigate the meaning of this warning and take steps to ensure the correct functionality.
### `info!()`
Use `-v` to enable.
This level is for the benefit of developers. It should be used for sporadic and infrequent messages. The same message should not "spam" the logs. The VM should be usable when this level of debugging is enabled and trying to use `stdin/stdout` and the logs are going to `stderr`.
### `debug!()`
Use `-vv` to enable.
For the most verbose of logging messages. It is acceptable to "spam" the log with repeated invocations of the same message. This level of logging would be combined with `--log-file`.
Cloud Hypervisor supports using a MACVTAP device which is derived from a MACVLAN. Full details of configuring MACVLAN or MACVTAP is out of scope of this document. However the example below indicates how to bridge the guest directly onto the the network the host is on. Due to the lack of hairpin mode it not usually possible to reach the guest directly from the host.
```bash
# The MAC address must be attached to the macvtap and be used inside the guest
mac="c2:67:4f:53:29:cb"
# Host network adapter to bridge the guest onto
host_net="eno1"
# Create the macvtap0 as a new virtual MAC associated with the host network
sudo ip link add link "$host_net" name macvtap0 type macvtap
sudo ip link set macvtap0 address "$mac" up
sudo ip link show macvtap0
# A new character device is created for this interface
tapindex=$(< /sys/class/net/macvtap0/ifindex)
tapdevice="/dev/tap$tapindex"
# Ensure that we can access this device
sudo chown "$UID.$UID""$tapdevice"
# Use --net fd=3 to point to fd 3 which the shell has opened to point to the /dev/tapN device
target/debug/cloud-hypervisor \
--kernel ~/src/linux/vmlinux \
--disk path=~/workloads/focal.raw \
--cpus boot=1 --memory size=512M \
--cmdline "root=/dev/vda1 console=hvc0"\
--net fd=3,mac=$mac 3<>$"$tapdevice"
```
As the guest is now connected to the same L2 network as the host you can obtain an IP address based on your host network (potentially including via DHCP)
This option is mandatory when using the `--memory` parameter.
Value is an unsigned integer of 64 bits.
_Example_
```
--memory size=1G
```
### `mergeable`
Specifies if the pages from the guest RAM must be marked as _mergeable_. In
case this option is `true` or `on`, the pages will be marked with `madvise(2)`
to let the host kernel know which pages are eligible for being merged by the
KSM daemon.
This option can be used when trying to reach a higher density of VMs running
on a single host, as it will reduce the amount of memory consumed by each VM.
By default this option is turned off.
_Example_
```
--memory size=1G,mergeable=on
```
### `shared`
Specifies if the memory must be `mmap(2)` with `MAP_SHARED` flag.
By sharing a memory mapping, one can share the guest RAM with other processes
running on the host. One can use this option when running vhost-user devices
as part of the VM device model, as they will be driven by standalone daemons
needing access to the guest RAM content.
By default this option is turned off, which results in performing `mmap(2)`
with `MAP_PRIVATE` flag.
_Example_
```
--memory size=1G,shared=on
```
### `hugepages` and `hugepage_size`
Specifies if the memory must be created and `mmap(2)` with `MAP_HUGETLB` and size
flags. This performs a memory mapping relying on the specified huge page size. If no huge page size is supplied the system's default huge page size is used.
By using hugepages, one can improve the overall performance of the VM, assuming
the guest will allocate hugepages as well. Another interesting use case is VFIO
as it speeds up the VM's boot time since the amount of IOMMU mappings are
reduced.
The user is responsible for ensuring there are sufficient huge pages of the specified size for the VMM to use. Failure to do so may result in strange VMM behaviour.
By default this option is turned off.
_Example_
```
--memory size=1G,hugepages=on,hugepage_size=2M
```
### `hotplug_method`
Selects the way of adding and/or removing memory to/from a booted VM.
Possible values are `acpi` and `virtio-mem`. Default value is `acpi`.
_Example_
```
--memory size=1G,hotplug_method=acpi
```
### `hotplug_size`
Amount of memory that can be dynamically added to the VM.
Value is an unsigned integer of 64 bits. A value of 0 is invalid.
_Example_
```
--memory size=1G,hotplug_size=1G
```
### `hotplugged_size`
Amount of memory that will be dynamically added to the VM at boot. This option
allows for starting a VM with a certain amount of memory that can be reduced
during runtime.
This is only valid when the `hotplug_method` is `virtio-mem` as it does not
make sense for the `acpi` use case. When using ACPI, the memory can't be
resized after it has been extended.
This option is only valid when `hotplug_size` is specified, and its value can't
exceed the value of `hotplug_size`.
Value is an unsigned integer of 64 bits. A value of 0 is invalid.
`MemoryZoneConfig` or what is known as `--memory-zone` from the CLI perspective
is a power user parameter. It allows for a full description of the guest RAM,
describing how every memory region is backed and exposed to the guest.
```rust
structMemoryZoneConfig{
id: String,
size: u64,
file: Option<PathBuf>,
shared: bool,
hugepages: bool,
host_numa_node: Option<u32>,
hotplug_size: Option<u64>,
hotplugged_size: Option<u64>,
}
```
```
--memory-zone <memory-zone> User defined memory zone parameters "size=<guest_memory_region_size>,file=<backing_file>,shared=on|off,hugepages=on|off,host_numa_node=<node_id>,id=<zone_identifier>,hotplug_size=<hotpluggable_memory_size>,hotplugged_size=<hotplugged_memory_size>"
```
This parameter expects one or more occurences, allowing for a list of memory
zones to be defined. It must be used with `--memory size=0`, clearly indicating
that the memory will be described through advanced parameters.
Each zone is given a list of options which we detail through the following
sections.
### `id`
Memory zone identifier. This identifier must be unique, otherwise an error will
be returned.
This option is useful when referring to a memory zone previously created. In
particular, the `--numa` parameter can associate a memory zone to a specific
NUMA node based on the memory zone identifier.
This option is mandatory when using the `--memory-zone` parameter.
Value is a string.
_Example_
```
--memory size=0
--memory-zone id=mem0,size=1G
```
### `size`
Size of the memory zone.
This option is mandatory when using the `--memory-zone` parameter.
Value is an unsigned integer of 64 bits.
_Example_
```
--memory size=0
--memory-zone id=mem0,size=1G
```
### `file`
Path to the file backing the memory zone. This can be either a file or a
directory. In case of a file, it will be opened and used as the backing file
for the `mmap(2)` operation. In case of a directory, a temporary file with no
hard link on the filesystem will be created. This file will be used as the
backing file for the `mmap(2)` operation.
This option can be particularly useful when trying to back a part of the guest
RAM with a well known file. In the context of the snapshot/restore feature, and
if the provided path is a file, the snapshot operation will not perform any
copy of the guest RAM content for this specific memory zone since the user has
access to it and it would duplicate data already stored on the current
filesystem.
Value is a string.
_Example_
```
--memory size=0
--memory-zone id=mem0,size=1G,file=/foo/bar
```
### `shared`
Specifies if the memory zone must be `mmap(2)` with `MAP_SHARED` flag.
By sharing a memory zone mapping, one can share part of the guest RAM with
other processes running on the host. One can use this option when running
vhost-user devices as part of the VM device model, as they will be driven
by standalone daemons needing access to the guest RAM content.
By default this option is turned off, which result in performing `mmap(2)`
with `MAP_PRIVATE` flag.
_Example_
```
--memory size=0
--memory-zone id=mem0,size=1G,shared=on
```
### `hugepages`
Specifies if the memory zone must be `mmap(2)` with `MAP_HUGETLB` and
`MAP_HUGE_2MB` flags. This performs a memory zone mapping relying on 2MiB
pages instead of the default 4kiB pages.
By using hugepages, one can improve the overall performance of the VM, assuming
the guest will allocate hugepages as well. Another interesting use case is VFIO
as it speeds up the VM's boot time since the amount of IOMMU mappings are
reduced.
By default this option is turned off.
_Example_
```
--memory size=0
--memory-zone id=mem0,size=1G,hugepages=on
```
### `host_numa_node`
Node identifier of a node present on the host. This option will let the user
pick a specific NUMA node from which the memory must be allocated. After the
memory zone is `mmap(2)`, the NUMA policy for this memory mapping will be
applied through `mbind(2)`, relying on the provided node identifier. If the
node does not exist on the host, the call to `mbind(2)` will fail.
This option is useful when trying to back a VM memory with a specific type of
memory from the host. Assuming a host has two types of memory, with one slower
than the other, each related to a distinct NUMA node, one could create a VM
with slower memory accesses by backing the entire guest RAM from the furthest
NUMA node on the host.
This option also gives the opportunity to create a VM with non uniform memory
accesses as one could define a first memory zone backed by fast memory, and a
second memory zone backed by slow memory.
Value is an unsigned integer of 32 bits.
_Example_
```
--memory size=0
--memory-zone id=mem0,size=1G,host_numa_node=0
```
### `hotplug_size`
Amount of memory that can be dynamically added to the memory zone. Since
`virtio-mem` is the only way of resizing a memory zone, one must specify
the `hotplug_method=virtio-mem` to the `--memory` parameter.
Value is an unsigned integer of 64 bits. A value of 0 is invalid.
_Example_
```
--memory size=0,hotplug_method=virtio-mem
--memory-zone id=mem0,size=1G,hotplug_size=1G
```
### `hotplugged_size`
Amount of memory that will be dynamically added to a memory zone at VM's boot.
This option allows for starting a VM with a certain amount of memory that can
be reduced during runtime.
This is only valid when the `hotplug_method` is `virtio-mem` as it does not
make sense for the `acpi` use case. When using ACPI, the memory can't be
resized after it has been extended.
This option is only valid when `hotplug_size` is specified, and its value can't
exceed the value of `hotplug_size`.
Value is an unsigned integer of 64 bits. A value of 0 is invalid.
`NumaConfig` or what is known as `--numa` from the CLI perspective has been
introduced to define a guest NUMA topology. It allows for a fine description
about the CPUs and memory ranges associated with each NUMA node. Additionally
it allows for specifying the distance between each NUMA node.
```rust
structNumaConfig{
id: u32,
cpus: Option<Vec<u8>>,
distances: Option<Vec<NumaDistance>>,
memory_zones: Option<Vec<String>>,
}
```
```
--numa <numa> Settings related to a given NUMA node "id=<node_id>,cpus=<cpus_id>,distances=<list_of_distances_to_destination_nodes>,memory_zones=<list_of_memory_zones>"
```
### `guest_numa_id`
Node identifier of a guest NUMA node. This identifier must be unique, otherwise
an error will be returned.
This option is mandatory when using the `--numa` parameter.
Value is an unsigned integer of 32 bits.
_Example_
```
--numa guest_numa_id=0
```
### `cpus`
List of virtual CPUs attached to the guest NUMA node identified by the
`guest_numa_id` option. This allows for describing a list of CPUs which
must be seen by the guest as belonging to the NUMA node `guest_numa_id`.
One can use this option for a fine grained description of the NUMA topology
regarding the CPUs associated with it, which might help the guest run more
efficiently.
Multiple values can be provided to define the list. Each value is an unsigned
integer of 8 bits.
For instance, if one needs to attach all CPUs from 0 to 4 to a specific node,
the syntax using `-` will help define a contiguous range with `cpus=0-4`. The
same example could also be described with `cpus=0:1:2:3:4`.
A combination of both `-` and `:` separators is useful when one might need to
describe a list containing all CPUs from 0 to 99 and the CPU 255, as it could
simply be described with `cpus=0-99:255`.
_Example_
```
--cpus boot=8
--numa guest_numa_id=0,cpus=1-3:7
--numa guest_numa_id=1,cpus=0:4-6
```
### `distances`
List of distances between the current NUMA node referred by `guest_numa_id`
and the destination NUMA nodes listed along with distances. This option let
the user choose the distances between guest NUMA nodes. This is important to
provide an accurate description of the way non uniform memory accesses will
perform in the guest.
One or more tuple of two values must be provided through this option. The first
value is an unsigned integer of 32 bits as it represents the destination NUMA
node. The second value is an unsigned integer of 8 bits as it represents the
distance between the current NUMA node and the destination NUMA node. The two
values are separated by `@` (`value1@value2`), meaning the destination NUMA
node `value1` is located at a distance of `value2`. Each tuple is separated
from the others with `:` separator.
For instance, if one wants to define 3 NUMA nodes, with each node located at
different distances, it can be described with the following example.
_Example_
```
--numa guest_numa_id=0,distances=1@15:2@25
--numa guest_numa_id=1,distances=0@15:2@20
--numa guest_numa_id=2,distances=0@25:1@20
```
### `memory_zones`
List of memory zones attached to the guest NUMA node identified by the
`guest_numa_id` option. This allows for describing a list of memory ranges
which must be seen by the guest as belonging to the NUMA node `guest_numa_id`.
This option can be very useful and powerful when combined with `host_numa_node`
option from `--memory-zone` parameter as it allows for creating a VM with non
uniform memory accesses, and let the guest know about it. It allows for
exposing memory zones through different NUMA nodes, which can help the guest
workload run more efficiently.
Multiple values can be provided to define the list. Each value is a string
referring to an existing memory zone identifier. Values are separated from
cloud-hypervisor can emulate one or more virtual network interfaces, represented at the hypervisor host by [tap devices](https://www.kernel.org/doc/Documentation/networking/tuntap.txt"). This guide briefly describes, in a manual and distribution neutral way, how to setup and use networking with cloud-hypevisor.
cloud-hypervisor can emulate one or more virtual network interfaces, represented at the hypervisor host by [tap devices](https://www.kernel.org/doc/Documentation/networking/tuntap.txt). This guide briefly describes, in a manual and distribution neutral way, how to setup and use networking with cloud-hypervisor.
## Multiple queue support for net devices ##
While multiple vcpus defined for guest, to gain the benefit of vcpu scalable to improve performance, it suggests to define multiple queue pairs for net devices, one Tx/Rx queue pair per one vcpu, that means the number of queue pairs at least is equal to the vcpu count. In that case, after virtnet driver set cpu affinity for virtqueues in guest kernel, vcpus could handle interrupt from different virtqueue pairs in parallel.
It will gain better performance for guest that has multiple queues defined for net devices while it has multiple net sessions running in userspace.
To enable multiple queue support in cloud-hypervisor, multiple queue pairs will be defined, while multiple tap fds will be opened for the same tap device, it will also have multiple threads started, each thread will monitor and handle the events from each virtqueue pairs and the associated tap fd.
Note:
- Currently, it does not support to use ethtool to change the combined queue numbers in guest.
- Multiple queue is enabled for vhost-user-net backend in cloud-hypervisor, however, multiple thread is not added to handle mq, thus, the performance for vhost-user-net backend is not supposed to be improved. The multiple thread will be added for backend later.
- Performance test for vhost-user-net will be covered once vhost-user-net backend has multiple thread supported.
- Performance test for virtio-net is done by comparing 2 queue pairs with 1 queue pairs, that to run 2 iperf3 sessions in the same test environments, throughput is improved about 37%.
## Start cloud-hypervisor with net devices
@@ -10,21 +25,37 @@ Use one `--net` command-line argument from cloud-hypervisor to specify the emula
num_queues is the total number of tx and rx queues, the default value is 2, and it could be increased by multiples of 2. Additionally, num_queues is suggested to be as 2 times of vcpu count. The default value for queue_size is 256.
If the tap device is pre-created on host before guest boot up. To use multiple queue support for net device in guest, the tap device should be opened like this from host.
```bash
[root@localhost ~]# ip tuntap add name ich0 mode tap multi_queue
```
And the `--net` device should specify support for multiple queues. `num_queues` must be a multiple of 2 starting at least from 4 since multiple queues really means multiple queue pairs. We need at least 2 pairs for this configuration to be correct:
`perf` can be used to profile the `cloud-hypervisor` binary but it is necessary to make some modifications to the the build in order to produce a binary that gives useful results.
## Building a suitable binary
Modify the `Cargo.toml` file to add `debug = 1` to the `[profile.release]` block. It should look like this:
```
[profile.release]
lto = true
debug = 1
```
This adds the symbol information to the release binary but does not otherwise affect the performance.
The binary must also be built with frame pointers included so that the call graph can be captured by the profiler.
$ perf record -g target/release/cloud-hypervisor \
--kernel ~/src/linux/vmlinux \
--pmem file=~/workloads/focal.raw \
--cpus boot=1 --memory size=1G \
--cmdline "root=/dev/pmem0p1 console=ttyS0" \
--serial tty --console off \
--api-socket=/tmp/api1
```
For analysing the samples:
```
$ perf report -g
```
If profiling with a network device attached either the TAP device must be already created and configured or the profiling must be done as root so that the TAP device can be created.
Cloud Hypervisor supports UEFI boot through the utilization of the EDK II based UEFI firmware.
## Building UEFI Firmware
To avoid any unnecessary issues, it is recommended to use Ubuntu 18.04 and its default toolset. Any other compatible Linux distribution is otherwise suitable, however it is suggested to use a temporary Docker container with Ubuntu 18.04 for a quick build on an existing Linux machine.
The commands below will compile an OVMF firmware suitable for Cloud Hypervisor.
After the successful build, the resulting firmware binaries are available under `Build/OvmfCh/DEBUG_GCC5/FV` underneath the edk2 checkout.
## Using OVMF Binaries
Any UEFI capable image can be booted using the Cloud Hypervisor specific firmware. Windows guests under Cloud Hypervisor only support UEFI boot, therefore OVMF is mandatory there.
To make Cloud Hypervisor use UEFI boot, pass the `OVMF.fd` file path as an argument to the `--kernel` option. The firmware file will be opened in read only mode.
The same firmware can be used with Cloud Hypervisor or with QEMU. This is particularly useful if using QEMU for the preparation phase.
## Building UEFI Firmware with Compatibility Support Module (CSM)
CSM is a module that allows to boot legacy operating systems using the OVMF firmware. OVMF can embed a CSM build of SeaBIOS. To build the SeaBIOS with CSM support, add `CONFIG_CSM=y` to `.config` before the build. The outcome `out/Csm16.bin` is to be moved into `OvmfPkg/Csm/Csm16/Csm16.bin` before OVMF is built. Then, the OVMF build will have to be passed the `-D CSM_ENABLE` option in order to generate a legacy aware UEFI firmware. At the current stage, all the necessary patches are included in the Cloud Hypervisor specific [SeaBIOS branch](https://github.com/cloud-hypervisor/seabios/tree/ch). Taking into account the previous instructions, the modified command sequence to compile an OVMF binary with CSM support is the following one:
Please note, that the CSM support has currently only been tested with Linux guests. There are no plans to provide legacy support for other OSes (e.g. Windows).
The test runs with multiple queue (MQ) support enabled, using 2 pairs of TX/RX queues defined for both OVS and the virtual machine. Here are the details on how the test can be run.
_Setup OVS_
`ovs_test.sh` is created to setup and start OVS. OVS will provide the `dpdkvhostuser` backend running in server mode.
Starting with the release version [0.10.0](https://github.com/cloud-hypervisor/cloud-hypervisor/releases/tag/v0.10.0), Cloud Hypervisor supports Windows guests.
__Requirements__
- Host with KVM enabled
- [UEFI](uefi.md) capable Windows guest image with Virtio drivers integrated
Any modern Windows Server version is compatible. Cloud Hypervisor has been successfully tested with Windows Server 2019 and Windows Server Core 2004.
At the current stage, only UEFI capable Windows images are supported. This implies the presence of the OVMF firmware during the Windows installation and in any subsequent usage. BIOS boot is not supported.
The subsequent sections will tell, in detail, how to prepare an appropriate Windows image.
## Image Preparation
### Installation using the stock Windows ISO
__Prerequisites__
- QEMU, version >=5.0.0 is recommended.
- Windows installation ISO. Obtained through MSDN, Visual Studio subscription, evaluation center, etc.
- With the suggested image size of 30G, there should be enough free disk space to hold the installation ISO and any other necessary files
This step currently requires QEMU to install Windows onto the guest. QEMU is only used at the preparation stage, the resulting image is then fully functional with Cloud Hypervisor.
Preparing several command parts as these will be used in the follow up sections as well.
Before the installation can proceed, point the Windows installation program to the VirtIO disk and install the necessary storage controller drivers. After that, the attached hard drive will become visible and the actual installation can commence.
After the installation has completed, proceed further to the configuration section. QEMU will be needed at least once more to enable the Windows Special Administration Console (SAC) and to possibly install extra device drivers.
## Image Usage
The basic command to boot a Windows image. The configuration section should be checked before executing it for the first time.
```shell
cloud-hypervisor \
--kernel ./$OVMF_DIR/OVMF.fd \
--disk path=./$IMG_FILE\
--cpus boot=1,kvm_hyperv=on \
--memory size=4G \
--serial tty \
--console off \
--net tap=
```
It is necessary to always:
- Carry the OVMF firmware in the `--kernel` option
- Add `kvm_hyperv=on` to the `--cpus` option
In cases where the host processor supports address space > 39 bits, it might be necessary to limit the address space. It can be done by appending the option `max_phys_bits=X` to the `--cpus` parameter, where `X` is the number of bits to be supported. Windows was tested to support at least 39-bit address space.
To daemonize the Cloud Hypervisor process, `nohup` can be used. Some STDIO redirections might need to be done. In a simple case it is sufficient to just redirect all the output to `/dev/null`.
## Image Configuration
### Device Drivers
After the Windows installation has finished under QEMU, there might be still devices with no drivers installed. This might happen for example, when a device was not used during the installation. In particular it is important to ensure that the VirtIO network device is setup correctly because further steps for the configuration and the usage require network in most case.
Boot once more under QEMU and use the [Device Manager](https://support.microsoft.com/en-in/help/4028443/windows-10-update-drivers), to ensure all the device drivers, and especially the network card, are installed correctly. Also, as Cloud Hypervisor can introduce new devices, it is advisable to repeat the procedure while booted under Cloud Hypervisor, when the RDP access to the image is functional.
### Windows Special Administration Console (SAC) enablement
SAC provides a text based console access to the Windows guest. As Cloud Hypervisor doesn't implement a VGA adaptor, SAC is an important instrument for the Windows guest management.
Boot the Windows image under QEMU and execute the below commands to permanently enable SAC
```cmd
bcdedit /emssettings emsport:1 emsbaudrate:115200
bcdedit /ems on
bcdedit /bootems on
```
Once SAC is enabled, the image can be booted under Cloud Hypervisor. The SAC prompt will show up
<pre>
Computer is booting, SAC started and initialized.
Use the "ch -?" command for information about using channels.
Use the "?" command for general help.
SAC>
</pre>
To open a console on the guest, the command sequence below can be used
<pre>
SAC>cmd
The Command Prompt session was successfully launched.
SAC>
EVENT: A new channel has been created. Use "ch -?" for channel help.
Channel: Cmd0001
SAC>ch -si 1
</pre>
See also the [links](#Links) section for a more extended SAC documentation.
## Network
This section illustrates the Windows specific corner points for the VM network configuration. For the extended networking guide, including bridging for multiple VMs, follow [networking.md](networking.md).
### Basic Networking
As the simplest option, using `--net tap=` in the Cloud Hypervisor command line will create a `vmtapX` device on the host with the default IPv4 adress `192.168.249.1`. After SAC becomes available, the guest configuration can be set with
This allows for SSH login from a remote machine, for example through the `administrator` user: `ssh administrator@192.168.249.2`. For a more detailed OpenSSH guide, please follow the MSDN article from the [links](#links) section.
## Debugging
The Windows guest debugging process relies heavily on QEMU and [socat](http://www.dest-unreach.org/socat/). The procedure requires two Windows VMs:
- A debugger VM running under QEMU.
- A debuggee, a Windows VM that has been created in the previous steps, running under Cloud Hypervisor or QEMU.
The connection between both guests happens over TCP, whereby on the guest side it is automatically translated to a COM port. Because the VMs are connected through TCP, the debugging infrastructure can be distributed over the network. The serial port, while slowly transferring data, is common enough to support a wide range of cases and tools.
In this excercise, [WinDbg](https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/) is used. Any other debugger of choice with the ability to use serial connection can be used instead.
### Debugger and Debuggee
#### WinDbg VM
For simplicity, the debugger VM is supposed to be only running under QEMU. It will require VGA and doesn't neccessarily depend on UEFI. As an OS, it can carry any supported Windows OS where the debugger of choice can be installed. The simplest way is to follow the image preparation instructions from the previous chapter, but avoid using the OVMF firmware. It is also not required to use VirtIO drivers, whereby it might be useful in some case. Though, while creating the image file for the debugger VM, be sure to choose a sufficient disk size that counts in the need to save the corresponding debug symbols and sources.
To create the debugger Windows VM, the following command can be used:
A non server Windows OS like Windows 10 can be used to carry the debugging tools in the debugger VM.
#### Debuggee VM
The debuggee VM is the one that we've learned to configure and run in the first section. There might be various reasons to debug. For example, there could be an issue in the Windows guest with an emulated device or an included driver. Or, we might want to develop a custom feature like a kernel driver to be available in the guest.
Note, that there are several ways to debug Windows, not all of them need to be enabled at the same time. For example, if developing a kernel module, the only useful options would be to configure for the serial debugging and enable the kernel debug. In that case, any crash or misbehavior in the boot loader or kernel would be ignored. The commands below must be run as administrator on the debuggee guest VM.
##### Turn On Serial Debugging
This will configure the debugging to be enabled and instruct to use the serial port for it.
```cmd
bcdedit /dbgsettings serial debugport:1 baudrate:115200
```
##### Turn On Kernel Debuging
```cmd
bcdedit /debug on
```
##### Turn On Boot Loader Debug
```cmd
bcdedit /bootdebug on
```
##### Turn on boot manager debug
```cmd
bcdedit /set {bootmgr} bootdebug on
```
##### Disable Recovery Screen On Boot Failure
There could be a situation, where a crash is debugged. In such cases, the guest could be left in an inconsistent state. The default Windows behavior would be to boot into the recovery screen, however in some cases it might be not desired. To make Windows ignore failures and always proceed to booting the OS, use the command below:
Note, this VM has the networking enabled. It is needed, because symbols and sources might need to be fetched from a network location.
Also, notice the `-serial` parameter - that's what does the magic on exposing the serial port to the guest while connecting the debugger VM with a client VM through the network. SAC/EMS needs to be disabled in the debugger VM, as otherwise the COM device might be blocked.
Hereafter, WinDbg can be started using a command below:
```cmd
set_NT_DEBUG_PORT=com1
set_NT_DEBUG_BAUD_RATE=115200
windbg -v -d -k
```
Once started, WinDbg will wait for an incoming connection which is going to be initialized by the debuggee VM started in the next section.
#### Invoke the Debuggee VM
##### Under QEMU
Essentially it would be the command like depicted in the guest preparation sections, with a few modifications:
It is to see, that `-serial` parameter is used here, to establish the connection with the debugger VM.
To disable HPET, attach `--no-hpet`. To enable hypervisor reference timer, use `-cpu host,hv-time`. These and other options can be used to achieve better [Hyper-V compatibility](https://archive.fosdem.org/2019/schedule/event/vai_enlightening_kvm/attachments/slides/2860/export/events/attachments/vai_enlightening_kvm/slides/2860/vkuznets_fosdem2019_enlightening_kvm.pdf).
##### Cloud Hypervisor
The `socat` tool is used to establish the QEMU compatible behavior. Here as well, the Cloud Hypervisor command used to run the Windows guest is to be used. Put the command into a shell script:
The reason to pack the command into the shell script is that the command might contain a comma. When using SYSTEM, the shell command can't contain `,` or `!!`.
## Links
- [Fedora VirtIO guide for Windows](https://docs.fedoraproject.org/en-US/quick-docs/creating-windows-virtual-machines-using-virtio-drivers/)
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
