DescriptorChain::is_valid() wrongly used .checked_offset() to attempt to
validate that the descriptor's data is in valid memory. This works in
all cases except where the guest has placed the data at the very end of
the guest memory as the offset + offset will be outside the range (as
the combined offset will be the next byte and as such out of the guest
memory). Instead use the function .check_range() takes an offset and a
length to validate
This fixes issues see with error messages featuring the
DescriptorChainTooShort error.
Fixes: #2424
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
(cherry picked from commit 1eb4ebe3d1)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
There are two parts:
- Unconditionally zero the output area. The length of the incoming
vector has been seen from 1 to 4 bytes, even though just the first
byte might need to be handled. But also, this ensures any possibly
unhandled offset will return zeroed result to the caller. The former
implementation used an I/O port which seems to behave differently from
MMIO and wouldn't require explicit output zeroing.
- An access with zero offset still takes place and needs to be handled.
Fixes#2437.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
(cherry picked from commit 9e9aba7c0b)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The following is from the Hyper-V specification v6.0b.
Cpuid leaf 0x40000003 EDX:
Bit 3: Support for physical CPU dynamic partitioning events is
available.
When Windows determines to be running under a hypervisor, it will
require this cpuid bit to be set to support dynamic CPU operations.
Cpuid leaf 0x40000004 EAX:
Bit 5: Recommend using relaxed timing for this partition. If
used, the VM should disable any watchdog timeouts that
rely on the timely delivery of external interrupts.
This bit has been figured out as required after seeing guest BSOD
when CPU hotplug bit is enabled. Race conditions seem to arise after a
hotplug operation, when a system watchdog has expired.
Closes#1799.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
(cherry picked from commit 5b168f54a6)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
error: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
Error: --> devices/src/legacy/rtc_pl031.rs:73:1
|
73 | impl Into<libc::clockid_t> for ClockType {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= note: `-D clippy::from-over-into` implied by `-D warnings`
= help: consider to implement `From` instead
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
error: redundant slicing of the whole range
Error: --> devices/src/legacy/gpio_pl061.rs:298:37
|
298 | let value = read_le_u32(&data[..]);
| ^^^^^^^^^ help: use the original slice instead: `data`
|
= note: `-D clippy::redundant-slicing` implied by `-D warnings`
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#redundant_slicing
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
error: name `GPIOInterruptDisabled` contains a capitalized acronym
Error: --> devices/src/legacy/gpio_pl061.rs:46:5
|
46 | GPIOInterruptDisabled,
| ^^^^^^^^^^^^^^^^^^^^^ help: consider making the acronym lowercase, except the initial letter: `GpioInterruptDisabled`
|
= note: `-D clippy::upper-case-acronyms` implied by `-D warnings`
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#upper_case_acronyms
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
warning: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
--> hypervisor/src/vm.rs:41:1
|
41 | impl Into<u64> for DataMatch {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= note: `#[warn(clippy::from_over_into)]` on by default
= help: consider to implement `From` instead
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into
warning: 1 warning emitted
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
warning: name `AcpiPMTimerDevice` contains a capitalized acronym
--> devices/src/acpi.rs:175:12
|
175 | pub struct AcpiPMTimerDevice {
| ^^^^^^^^^^^^^^^^^ help: consider making the acronym lowercase, except the initial letter: `AcpiPmTimerDevice`
|
= note: `#[warn(clippy::upper_case_acronyms)]` on by default
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#upper_case_acronyms
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
warning: name `IORegion` contains a capitalized acronym
--> pci/src/configuration.rs:320:5
|
320 | IORegion = 0x01,
| ^^^^^^^^ help: consider making the acronym lowercase, except the initial letter (notice the capitalization): `IoRegion`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#upper_case_acronyms
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
warning: name `ConvertFromUTF8` contains a capitalized acronym
--> virtio-devices/src/vsock/unix/mod.rs:32:5
|
32 | ConvertFromUTF8(std::str::Utf8Error),
| ^^^^^^^^^^^^^^^ help: consider making the acronym lowercase, except the initial letter: `ConvertFromUtf8`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#upper_case_acronyms
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
warning: name `InvalidCPUCount` contains a capitalized acronym
--> src/bin/ch-remote.rs:24:5
|
24 | InvalidCPUCount(std::num::ParseIntError),
| ^^^^^^^^^^^^^^^ help: consider making the acronym lowercase, except the initial letter: `InvalidCpuCount`
|
= note: `#[warn(clippy::upper_case_acronyms)]` on by default
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#upper_case_acronyms
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
warning: name `TranslateGVA` contains a capitalized acronym
--> hypervisor/src/arch/emulator/mod.rs:51:5
|
51 | TranslateGVA(#[source] anyhow::Error),
| ^^^^^^^^^^^^ help: consider making the acronym lowercase, except the initial letter: `TranslateGva`
|
= note: `#[warn(clippy::upper_case_acronyms)]` on by default
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#upper_case_acronyms
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
error: redundant slicing of the whole range
--> vhost_user_block/src/lib.rs:396:31
|
396 | right.copy_from_slice(&data[..]);
| ^^^^^^^^^ help: use the original slice instead: `data`
|
= note: `-D clippy::redundant-slicing` implied by `-D warnings`
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#redundant_slicing
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
error: name `RSDPPastRamEnd` contains a capitalized acronym
--> arch/src/lib.rs:59:5
|
59 | RSDPPastRamEnd,
| ^^^^^^^^^^^^^^ help: consider making the acronym lowercase, except the initial letter: `RsdpPastRamEnd`
|
= note: `-D clippy::upper-case-acronyms` implied by `-D warnings`
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#upper_case_acronyms
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
error: writing `&PathBuf` instead of `&Path` involves a new object where a slice will do.
--> block_util/src/lib.rs:68:31
|
68 | fn build_device_id(disk_path: &PathBuf) -> result::Result<String, Error> {
| ^^^^^^^^ help: change this to: `&Path`
|
= note: `-D clippy::ptr-arg` implied by `-D warnings`
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#ptr_arg
error: writing `&PathBuf` instead of `&Path` involves a new object where a slice will do.
--> block_util/src/lib.rs:83:39
|
83 | pub fn build_disk_image_id(disk_path: &PathBuf) -> Vec<u8> {
| ^^^^^^^^ help: change this to: `&Path`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#ptr_arg
error: writing `&PathBuf` instead of `&Path` involves a new object where a slice will do.
--> block_util/src/lib.rs:68:31
|
68 | fn build_device_id(disk_path: &PathBuf) -> result::Result<String, Error> {
| ^^^^^^^^ help: change this to: `&Path`
|
= note: `-D clippy::ptr-arg` implied by `-D warnings`
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#ptr_arg
error: writing `&PathBuf` instead of `&Path` involves a new object where a slice will do.
--> block_util/src/lib.rs:83:39
|
83 | pub fn build_disk_image_id(disk_path: &PathBuf) -> Vec<u8> {
| ^^^^^^^^ help: change this to: `&Path`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#ptr_arg
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
error: redundant slicing of the whole range
--> net_util/src/mac.rs:60:35
|
60 | bytes[..].copy_from_slice(&src[..]);
| ^^^^^^^^ help: use the original slice instead: `src`
|
= note: `-D clippy::redundant-slicing` implied by `-D warnings`
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#redundant_slicing
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
error: name `TYPE_UNKNOWN` contains a capitalized acronym
--> vm-virtio/src/lib.rs:48:5
|
48 | TYPE_UNKNOWN = 0xFF,
| ^^^^^^^^^^^^ help: consider making the acronym lowercase, except the initial letter: `Type_Unknown`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#upper_case_acronyms
error: name `GetDeviceID` contains a capitalized acronym
--> block_util/src/lib.rs:138:5
|
138 | GetDeviceID,
| ^^^^^^^^^^^ help: consider making the acronym lowercase, except the initial letter: `GetDeviceId`
|
= note: `-D clippy::upper-case-acronyms` implied by `-D warnings`
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#upper_case_acronyms
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
With CONFIG_PVH in stable kernels for some time we should deprecate the
use of alternative boot methods since this will lead to a much simpler
boot flow and CI process.
See: #2231
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
There are multiple reports of DescriptorChainTooShort errors and so add
some extra debugging to aid the debugging of this issue.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Drop the generic type E and use IrqRoutngEntry directly. This allows
dropping a bunch of trait bounds from code.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Their make_entry functions look the same now. Extract the logic to a
common function.
No functional change.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
This returns trues if this descriptor has another descriptor linked to
it. Not whether this descriptor chain has another one following it.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The 0.3.0 release (re)supports booting stock Ubuntu images so users
should be pointed towards that.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
There is no point in queueing an empty descriptor in the list of iovecs.
Let's simply ignore such case and avoid some unnecessary processing.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The hotplug documentation was missing the explanation on how to use
memory hotplug with virtio-mem.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Document that seccomp is on and how to disable it along with a pointer
on how to identify missing syscalls during development.
Fixes#993
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
It must be specified as excluded from the workspace as it must not be
built on non-test targets due to issues with the ssh2 dependency and the
musl toolchain.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
refactor vec_with_array_field to common hypervisor code
so that mshv can also make use of it.
Signed-off-by: Vineeth Pillai <viremana@linux.microsoft.com>
Bumps iced-x86 from 1.10.3 to 1.11.0.
Manual update of the code was needed since memory_displacement() was
deprecated and replaced with either memory_displacement32() or
memory_displacement64().
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
There's no need to have the code creating the passthrough_device being
duplicated since we can factorize it in a function used in both cases
(both cold plugged and hot plugged devices VFIO devices).
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Extend and use the existing DeviceTree to retrieve useful information
related to PCI devices. This removes the duplication with pci_devices
field which was internal to the DeviceManager.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Make the code a bit clearer by changing the naming of the structure
holding the list of IRQs reserved for PCI devices. It is also modified
into an array of 32 entries since we know this is the amount of PCI
slots that is supported.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We define a new enum in order to classify PCI device under virtio or
VFIO. This is a cleaner approach than using the Any trait, and
downcasting it to find the object back.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Introduces a tuple holding both information needed by pci_id_list and
pci_devices.
Changes pci_devices to be a BTreeMap of this new tuple.
Now that pci_devices holds the information needed from pci_id_list,
pci_id_list is no longer needed.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In anticipation for further factorization, the pci_id_list is now a
hashmap of PCI b/d/f leading to each device name.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Instead of relying on a PCI specific device list, we use the DeviceTree
as a reference to determine if a device name is already in use or not.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Remove the startup message containing incomplete VM configuration
details. It's a bit unusual for a tool such as Cloud Hypervisor to print
those kind of details which are a direct representation of the command
line.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This includes:
* OS disk image management
* Cloud init creation
* SSH to guest access
* Waiting for guest to boot
This will be useful in other projects that want to do similar things in
their integration tests.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Only if we have a valid API server path then create the API server. For
now this has no functional change there is a default API server path in
the clap handling but rather prepares to do so optionally.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In case of the virtio frontend driver doesn't need interupts for
certain queue event, it may explicitly write VIRTIO_MSI_NO_VECTOR
to the virtio common configuration, or it may doesn't configure
the event type vector at all.
This patch initializes both MSI-X configuration vector and queue vector
with VIRTIO_MSI_NO_VECTOR, so that the backend drivers won't trigger
unexpected interrupts to the guest.
Signed-off-by: Zide Chen <zide.chen@intel.com>
Relying on a NVIDIA Tesla T4 card present in the SGX machine, this patch
enables baremetal VFIO testing, validated by running several NVIDIA
tools in the guest. The guest image has been prepared to include all the
software needed to run these tests.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit implements ARM PrimeCell General Purpose Input/Output
(GPIO) PL061 device specification.
Signed-off-by: Penny Zheng <penny.zheng@arm.com>
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This commit switches the default serial device from 16550 to the
Arm dedicated UART controller PL011. The `ttyAMA0` can be enabled.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Enabled all "ttyS0" related test cases:
- test_serial_off
- test_serial_tty
- test_serial_file
Enabled mandatory guest kernel driver for "ns16550a" on AArch64.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
On AArch64, interrupt controller (GIC) is emulated by KVM. VMM need to
set IRQ routing for devices, including legacy ones.
Before this commit, IRQ routing was only set for MSI. Legacy routing
entries of type KVM_IRQ_ROUTING_IRQCHIP were missing. That is way legacy
devices (like serial device ttyS0) does not work.
The setting of X86 IRQ routing entries are not impacted.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Extend the existing url_to_path() to take the URL string and then use
that to simplify the snapshot/restore code paths.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Now that virtio devices can be updated with add_memory_region(), there's
no need to keep update_memory() around.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Relies on the preliminary work allowing virtio devices to be updated
with a single memory at a time instead of updating the entire memory at
once.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Assuming vhost-user devices support CONFIGURE_MEM_SLOTS protocol
feature, we introduce a new method to the VirtioDevice trait in order to
update one single memory at a time.
In case CONFIGURE_MEM_SLOTS is not supported by the backend (feature not
acked), we fallback onto the current way of updating the memory
mappings, that is with SET_MEM_TABLE.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to support GET_MAX_MEM_SLOTS, ADD_MEM_REG and REM_MEM_REG, the
protocol feature CONFIGURE_MEM_SLOTS must be enabled.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to support GET_MAX_MEM_SLOTS, ADD_MEM_REG and REM_MEM_REG, the
protocol feature CONFIGURE_MEM_SLOTS must be enabled.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The latest vhost version adds the support for the new commands
get_max_mem_slots(), add_mem_region() and remove_mem_region(), all
related to the new vhost-user protocol feature CONFIGURE_MEM_SLOTS.
The vhost_user_backend crate is updated accordingly in order to support
these new commands, mostly related to the capability of updating the
guest memory mappings with a finer control than set_mem_table() command.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
On x86_64 architecture, multiple syscalls were missing when shutting
down the vhost-user-net device along with the VM. This was causing the
usual crash related to seccomp filters.
This commit adds these missing syscalls to fix the issue.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
There is no need to get the vring base when resetting the vhost-user
device. This was mostly ignored, but in some cases, it was causing some
actual errors.
A reset must simply be a combination of disabling the vrings along with
the reset of the owner.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Originally, VhostUserSetup is only used by vhost-user-fs. While
vhost-user-blk and vhost-user-net have their own error messages,
we rename VhostUserSetup to VhostUserFsSetup.
Signed-off-by: Jiachen Zhang <zhangjiachen.jaycee@bytedance.com>
The MMIO structure contains the length rather than the maximum address
so it is necessary to subtract the starting address from the end address
to calculate the length.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In the absence of a way of integration testing this testing that it
compiles is reasonable compromise.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Load the sections backed from the file into their required addresses in
memory and populate the HOB with details of the memory. Using the HOB
address initialize the TDX state in the vCPUs and finalize the TDX
configuration.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add API to the hypervisor interface and implement for KVM to allow the
special TDX KVM ioctls on the VM and vCPU FDs.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When booting with TDX no kernel is supplied as the TDFV is responsible
for loading the OS. The requirement to have the kernel is still
currently enforced at the validation entry point; this change merely
changes function prototypes and stored state to use Option<> to support.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add support extracting the sections out for a TDVF file which can be
then used to load the TDVF and TD HOB data into their appropriate
locations.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add the skeleton of the "tdx" feature with a module ready inside the
arch crate to store implementation details.
TEST=cargo build --features="tdx"
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
These need to be updated together as the kvm-ioctls depends upon a
strictly newer version of kvm-bindings which requires a rebase in the CH
fork.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When a vm is created with a pty device, on reboot the pty fd (sub
only) will only be associated with the vmm through the epoll event
loop. The fd being polled will have been closed due to the vm itself
dropping the pty files (and potentially reopening the fd index to a
different item making things quite confusing) and new pty fds will be
opened but not polled on for input.
This change creates a structure to encapsulate the information about
the pty fd (main File, sub File and the path to the sub File). On
reboot, a copy of the console and serial pty structs is then passed
down to the new Vm instance which will be used instead of creating a
new pty device.
This resolves the underlying issue from #2316.
Signed-off-by: William Douglas <william.r.douglas@gmail.com>
Now that virtio-mem devices can update VFIO mappings through dedicated
handlers, let's provide them from the DeviceManager.
Important to note these handlers should either be provided to virtio-mem
devices or to the unique virtio-iommu device. This must be mutually
exclusive.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Create two functions for registering/unregistering DMA mapping handlers,
each handler being associated with a VFIO device.
Whenever the plugged_size is modified (which means triggered by the
virtio-mem driver in the guest), the virtio-mem backend is responsible
for updating the DMA mappings related to every VFIO device through the
handler previously provided.
It's important to update the map when the handler is either registered
or unregistered as well, as we don't want to miss some plugged memory
that would have been added before the VFIO device is added to the VM.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Instead of letting the VfioPciDevice take the decision on how/when to
perform the DMA mapping/unmapping, we move this to the DeviceManager
instead.
The point is to let the DeviceManager choose which guest memory regions
should be mapped or not. In particular, we don't want the virtio-mem
region to be mapped/unmapped as it will be virtio-mem device
responsibility to do so.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
When memory is resized through ACPI, a new region is added to the guest
memory. This region must also be added to the corresponding memory zone
in order to keep everything in sync.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Replace "--monitor-fd" with "--event-monitor" which can either take
"fd=<int>" or "path=<path>" which can point to e.g. a named pipe and
allow more flexibility.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Since the SGX server is down for maintenance, all builds are waiting on
the node agent to answer, causing all PRs to be blocked.
Let's disable temporarily the SGX CI until the server is back up.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
So far we've only had the need to emulate one instruction. There is no
need to use a HashMap when a simple tuple for the initial mapping will
do.
We can bring back the HashMap once more sophisticated use cases surface.
No functional change.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
This avoids code complexity down the line when we get around
implementing Windows support.
No functional change.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Moving to the latest version of the rust-vmm/vhost crate, before it gets
published on crates.io.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In particular update for the vmm-sys-util upgrade and all the other
dependent packages. This requires an updated forked version of
kvm-bindings (due to updated vfio-ioctls) but allowed the removal of our
forked version of kvm-ioctls.
The changes to the API from kvm-ioctls and vmm-sys-util required some
other minor changes to the code.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The vhost crate from rust-vmm is ready, which is why we do the switch
from the Cloud Hypervisor fork to the upstream crate.
At the same time, we rename the crate from vhost_rs to vhost.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit moves both pci and vmm code from the internal vfio-ioctls
crate to the upstream one from the rust-vmm project.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that ExternalDmaMapping is defined in vm-device, let's use it from
there.
This commit also defines the function get_host_address_range() to move
away from the vfio-ioctls dependency.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By moving the trait and its VFIO implementation out of vfio-ioctls, we
give anticipate for the move to the vfio-ioctls from rust-vmm.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This has been lagging behind on an older Rust version and we have
enough coverage from our Jenkins CI agent.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This is only use in the integration test and was erroneously included in
the main binary dependencies.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The interrupt tests were not being run as they were erroneously under a
feature guard that does not exist in arch.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This removes the dependency on "tempdir" which in turn depends on the
large rand dependency chain.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The help information displayed for our `--disk` option is incorrect and
incomplete, e.g. missing the `direct` and `poll_queue` field.
Signed-off-by: Bo Chen <chen.bo@intel.com>
According to Intel's mnemonic (which is used by iced-x86) the first
argument is destination while the second is source.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
The main idea behind this commit is to remove all the complexity
associated with TX/RX handling for virtio-net. By using writev() and
readv() syscalls, we could get rid of intermediate buffers for both
queues.
The complexity regarding the TAP registration has been simplified as
well. The RX queue is only processed when some data are ready to be
read from TAP. The event related to the RX queue getting more
descriptors only serves the purpose to register the TAP file if it's not
already.
With all these simplifications, the code is more readable but more
performant as well. We can see an improvement of 10% for a single
queue device.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Currently script does not exit early if the image/firmware not present
in the host. We should not progress further if the images are not pre
downloaded.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
This crate exposes the abililty for the VMM to set a file that events
should be written to. The event!() macro provides an interface to report
those events allowing the specification of an event source, an event
type and optional extra data. This will be written to the provided file
descriptor as JSON data.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
We don't have an easy way to figure out if a GPA points to normal memory
or device memory, but the guest's normal memory regions shouldn't
overlap with device regions. We can simply try to do a normal memory
read / write, and proceed to do device memory read / write if that
fails.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
This function can then be used by the TDX code to allocate the memory at
specific locations required for the TDVF to run from.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
OVMF would use string IO on those ports. String IO has not been
implemented, so that leads to panics.
Skip them explicitly in MSHV. Leave a long-ish comment in code to
explain the situation. We should properly implement string IO once it
becomes feasible / necessary.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Update for clippy in Rust 1.50.0:
error: Unnecessary nested match
--> vmm/src/vm.rs:419:17
|
419 | / if let vm_device::BusError::MissingAddressRange = e {
420 | | warn!("Guest MMIO write to unregistered address 0x{:x}", gpa);
421 | | }
| |_________________^
|
= note: `-D clippy::collapsible-match` implied by `-D warnings`
help: The outer pattern can be modified to include the inner pattern.
--> vmm/src/vm.rs:418:17
|
418 | Err(e) => {
| ^ Replace this binding
419 | if let vm_device::BusError::MissingAddressRange = e {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ with this pattern
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#collapsible_match
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
If the function can never return an error this is now a clippy failure:
error: this function's return value is unnecessarily wrapped by `Result`
--> virtio-devices/src/watchdog.rs:215:5
|
215 | / fn set_state(&mut self, state: &WatchdogState) -> io::Result<()> {
216 | | self.common.avail_features = state.avail_features;
217 | | self.common.acked_features = state.acked_features;
218 | | // When restoring enable the watchdog if it was previously enabled. We reset the timer
... |
223 | | Ok(())
224 | | }
| |_____^
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#unnecessary_wraps
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This addresses a newly added clippy issue:
error: methods called `as_*` usually take self by reference or self by mutable reference; consider choosing a less ambiguous name
--> vm-migration/src/protocol.rs:57:34
|
57 | fn as_mut_bytes<T: Sized>(p: &mut T) -> &mut [u8] {
| ^^^^^^
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#wrong_self_convention
error: aborting due to 2 previous errors
error: could not compile `vm-migration`
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Depending on the host OS the code for looking up the time for the CMOS
make require extra syscalls to be permitted for the vCPU thread.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In case the VFIO device does not support MSI or MSI-X, the capabilities
should not be parsed, avoiding the exposure of unsupported capabilities.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Make sure to propagate the error coming from VfioDevice when trying to
enable INTx, MSI or MSI-X interrutps.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Since INTx is now supported for VFIO PCI devices, there is no more
limitations regarding VFIO, which means the documentation must be
updated accordingly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
With all the preliminary work done in the previous commits, we can
update the VFIO implementation to support INTx along with MSI and MSI-X.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Here we are adding the PCI routing table, commonly called _PRT, to the
ACPI DSDT. For simplification reasons, we chose not to implement PCI
links as this involves dynamic decision from the guest OS, which result
in lots of complexity both from an AML perspective and from a device
manager perspective.
That's why the _PRT creates a static list of 32 entries, each assigned
with the IRQ number previously reserved by the device manager.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to support INTx for PCI devices, each PCI device must be
assigned an IRQ. This is preliminary work to reserve 8 IRQs which will
be shared across the 32 PCI devices.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In anticipation for accessing the legacy interrupt manager from the
function creating a VFIO PCI device, we store it as part of the
DeviceManager, to make it available for all methods.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The DeviceManager already has a hold onto the MSI interrupt manager,
therefore there's no need to pass it through every function. Instead,
let's simplify the code by using the attribute from DeviceManager's
instance.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Both GIC and IOAPIC must implement a new method notifier() in order to
provide the caller with an EventFd corresponding to the IRQ it refers
to.
This is needed in anticipation for supporting INTx with VFIO PCI
devices.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In anticipation for supporting the notifier function for the legacy
interrupt source group, we need this function to return an EventFd
instead of a reference to this same EventFd.
The reason is we can't return a reference when there's an Arc<Mutex<>>
involved in the call chain.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Swap the last two parameters of guest_mem_{read,write} to be consistent
with other read / write functions.
Use more descriptive parameter names.
No functional change.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
This reflects that it generates CPUID state used across all vCPUs.
Further ensure that errors from this function get correctly propagated.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Move the code for populating the CPUID with KVM HyperV emulation details from
the per-vCPU CPUID handling code to the shared CPUID handling code.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Move the code for populating the CPUID with details of the CPU
identification from the per-vCPU CPUID handling code to the shared CPUID
handling code.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Move the code for populating the CPUID with details of the maximum
address space from the per-vCPU CPUID handling code to the shared CPUID
handling code.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add the ability for cloud-hypervisor to create, manage and monitor a
pty for serial and/or console I/O from a user. The reasoning for
having cloud-hypervisor create the ptys is so that clients, libvirt
for example, could exit and later re-open the pty without causing I/O
issues. If the clients were responsible for creating the pty, when
they exit the main pty fd would close and cause cloud-hypervisor to
get I/O errors on writes.
Ideally the main and subordinate pty fds would be kept in the main
vmm's Vm structure. However, because the device manager owns parsing
the configuration for the serial and console devices, the information
is instead stored in new fields under the DeviceManager structure
directly.
From there hooking up the main fd is intended to look as close to
handling stdin and stdout on the tty as possible (there is some future
work ahead for perhaps moving support for the pty into the
vmm_sys_utils crate).
The main fd is used for reading user input and writing to output of
the Vm device. The subordinate fd is used to setup raw mode and it is
kept open in order to avoid I/O errors when clients open and close the
pty device.
The ability to handle multiple inputs as part of this change is
intentional. The current code allows serial and console ptys to be
created and both be used as input. There was an implementation gap
though with the queue_input_bytes needing to be modified so the pty
handlers for serial and console could access the methods on the serial
and console structures directly. Without this change only a single
input source could be processed as the console would switch based on
its input type (this is still valid for tty and isn't otherwise
modified).
Signed-off-by: William Douglas <william.r.douglas@gmail.com>
This thread is virtio-net specific, so it is not handled in the common
virtio device code.
The non-vhost implementation resumes the thread itself. Do the same
thing for vhost-user-net.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Use the newly added hugepages_size option if provided by the user to
pick a huge page size when creating the memfd region. If none is
specified use the system default.
Sadly different huge pages cannot be tested by an integration test as
creating a pool of the non-default size cannot be done at runtime
(requires kernel to be booted with certain parameters.)
TETS=Manually tested with a kernel booted with both 1GiB and 2MiB huge
pages (hugepagesz=1G hugepages=1 hugepagesz=2M hugepages=512)
Fixes: #2230
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Including a warning that the user is respsonsible for ensuring that they
have sufficient pages of the specified size.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This allows the user to use an alternative huge page size otherwise the
default size will be used.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This commit introduces a new information to the VirtioMemZone structure
in order to know if the memory zone is backed by hugepages.
Based on this new information, the virtio-mem device is now able to
determine if madvise(MADV_DONTNEED) should be performed or not. The
madvise documentation specifies that MADV_DONTNEED advice will fail if
the memory range has been allocated with some hugepages.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Signed-off-by: Hui Zhu <teawater@antfin.com>
This commit performs some refactoring to make all functions a method
from a specific object, and in particular methods for MemEpollHandler.
The point is to simplify the code to make it more readable.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Adjust the code to comply better with the virtio-mem specification by
adding some validation for the virtio-mem configuration, but also by
updating the virtio-mem configuration itself.
Nowhere in the virtio-mem specification is stated the usable region size
must be adjusted everytime the plugged size changes. For simplification
reason, and without going against the specification, the usable region
size is now kept static, setting its value to the size of the whole
region.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By introducing a ResizeSender object, we avoid having a Resize clone
with a different content than the original Resize object.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
MOV R/RM is a special case of MOVZX, so we generalize the mov_r_rm macro
to make it support both instructions.
Fixes: #2227
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The vhd module is the implementation of the VHD specification, which is
why it is important to unit test it.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Let's create a fixed VHD disk file from the existing RAW file thanks to
qemu-img, and create a new integration test to validate that
Cloud-Hypervisor can boot VHD disk image.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Relying on the simplified version of the synchronous support for RAW
disk files, the new fixed_vhd_sync module in the block_util crate
introduces the synchronous support for fixed VHD disk files.
With this patch, the fixed VHD support is complete as it is implemented
in both synchronous and asynchronous versions.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Using directly preadv and pwritev, we can simply use a RawFd instead of
a file, and we don't need to use the more complex implementation from
the qcow crate.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit adds the asynchronous support for fixed VHD disk files.
It introduces FixedVhd as a new ImageType, moving the image type
detection to the block_util crate (instead of qcow crate).
It creates a new vhd module in the block_util crate in order to handle
VHD footer, following the VHD specification.
It creates a new fixed_vhd_async module in the block_util crate to
implement the asynchronous version of fixed VHD disk file. It relies on
io_uring.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
_EJx built in should not return.
dsdt.dsl 813: Return (CEJ0 (0x00))
Warning 3104 - ^ Reserved method should not return a value (_EJ0)
dsdt.dsl 813: Return (CEJ0 (0x00))
Error 6080 - ^ Called method returns no value
Fixes: #2216
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The mutex timeout should be 0xffff rather than 0xfff to disable the
timeout feature.
dsdt.dsl 745: Acquire (\_SB.PRES.CPLK, 0x0FFF)
Warning 3130 - ^ Result is not used, possible operator timeout will be missed
dsdt.dsl 767: Acquire (\_SB.PRES.CPLK, 0x0FFF)
Warning 3130 - ^ Result is not used, possible operator timeout will be missed
dsdt.dsl 775: Acquire (\_SB.PRES.CPLK, 0x0FFF)
Warning 3130 - ^ Result is not used, possible operator timeout will be missed
Fixes: #2216
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
By using `net_util::open_tap` to create the TAP interface, the created
interface will be deleted when the returned variable (`net_utils::Tap`)
is dropped.
Signed-off-by: Bo Chen <chen.bo@intel.com>
This patch enables multi-queue support for creating virtio-net devices by
accepting multiple TAP fds, e.g. '--net fds=3:7'.
Fixes: #2164
Signed-off-by: Bo Chen <chen.bo@intel.com>
This helper can open a TAP device and configure the interface on it. If
the device needs to be opened multiple times for MQ then it also handles
that correctly.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Because of the behavior of the NVIDIA proprietary driver, we can't
expect NVIDIA cards with only MSI support to be functioning correctly
after they've been passed through with Cloud-Hypervisor.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Building with 1.51 nightly produces the following warning:
warning: unnecessary trailing semicolon
--> vmm/src/device_manager.rs:396:6
|
396 | };
| ^ help: remove this semicolon
|
= note: `#[warn(redundant_semicolons)]` on by default
warning: 1 warning emitted
Signed-off-by: Wei Liu <liuwe@microsoft.com>
The unit tests ask the Linux kernel to generate a TAP device name on
demand by passing in a format string. I suspect, but haven't been able
to confirm that there might be a rare race that triggers when creating
lots of devices in a short period of time. This is appearing in our unit
test as the occassional flake of the test_tap_read() which although it
has successfully created the device it fails to set the IP address on it
when looking it back up by it's name.
Since this is the most frequent cause of failures on our CI use a lock
to ensure that multiple TAP devices are not created simultaneously.
Fixes: #2135
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This skeleton commit brings in the support for compiling aarch64 with
the "acpi" feature ready to the ACPI enabling. It builds on the work to
move the ACPI hotplug devices from I/O ports to MMIO and conditionalises
any code that is x86_64 only (i.e. because it uses an I/O port.)
Filling in the aarch64 specific details in tables such as the MADT it
out of the scope.
See: #2178
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
e.g.
scripts/dev_cli.sh tests --integration -- --test-filter test_watchdog
This used to be supported by passing "$@" but was broken when multiple
hypervisor support was added.
Fixes: #2182
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Since QCOW and RAW synchronous implementation are very close, it makes
sense to introduce some common functions that can be shared between
these two.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
It might be useful debugging information for the user to know what kind
of disk file implementation is in use.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that BlockIoUring is the only implementation of virtio-block,
handling both synchronous and asynchronous backends based on the
AsyncIo trait, we can rename it to Block.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that both synchronous and asynchronous backends rely on the
asynchronous version of virtio-block (namely BlockIoUring), we can
get rid of the synchronous version (namely Block).
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the synchronous QCOW file implementation present in the qcow
crate, we created a new qcow_sync module in block_util that ports this
synchronous implementation to the AsyncIo trait.
The point is to reuse virtio-blk asynchronous implementation for both
synchronous and asynchronous backends.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the synchronous RAW file implementation present in the qcow
crate, we created a new raw_sync module in block_util that ports this
synchronous implementation to the AsyncIo trait.
The point is to reuse virtio-blk asynchronous implementation for both
synchronous and asynchronous backends.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the new DiskFile and AsyncIo traits, the implementation of
asynchronous block support does not have to be tied to io_uring anymore.
Instead, the only thing the virtio-blk implementation knows is that it
is using an asynchronous implementation of the underlying disk file.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Both DiskFile and AsyncIo traits are introduced to allow all kind of
files (RAW, QCOW, VHD) to be able to handle asynchronous access to the
underlying file.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Currently the GED control is in a fixed I/O port address but instead use
an MMIO address that has been chosen by the allocator.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This patch refines the sccomp filter list for the vCPU thread, as we are
no longer spawning virtio-device threads from the vCPU thread.
Fixes: #2170
Signed-off-by: Bo Chen <chen.bo@intel.com>
Even though the driver can provide fewer queues than those advertised
for some device types their is a minimum number that is required for
operation.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
It is permissable for the driver to program fewer queues than offered by
the device. Filter the queues so that only the ready ones are included
and check that they have valid addresses configured.
Fixes: #2136
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Don't assume that the number of queues provided match the number of
queues offered. The virtio spec allows the driver to program fewer
queues.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
We should use full memory barrier to ensure both guest and us
can see the correct avail_idx and avail_event_idx. Something
like this pattern:
VM: CLH:
update vring.avail->idx update avail_event_idx
mb() mb()
read avail_event_idx read vring.avail->idx
Signed-off-by: Xie Yongji <xieyongji@bytedance.com>
Rather than having to give and return the ioeventfd used for a device
clone them each time. This will make it simpler when we start handling
the driver enabling fewer queues than advertised by the device.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
We have killed the device thread by writing to the exit EventFd but we
should wait for them to quit to ensure consistency.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This will lead to the triggering of an ACPI button inside the guest in
order to cleanly shutdown the guest.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Use the ACPI GED device to trigger a notitifcation of type
POWER_BUTTON_CHANGED which will ultimately lead to the guest being
notified.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
If a GED event of type 0x8 (for power button) is received notify the
guest OS that this button device has been activated.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Renamed this bitfield as it will also be used for non-hotplug purposes
such as synthesising a power button.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In order to make the thread naming more useful derive their name from
the device id (which can be supplied by the user) and a device specific
suffix that has details of the individual queue (or queue pair.)
e.g.
rob@artemis:~$ pstree -p -c -l -t `pidof cloud-hypervisor`
cloud-hyperviso(27501)─┬─{_console}(27525)
├─{_disk0_q0}(27529)
├─{_disk0_q1}(27532)
├─{_net1_ctrl}(27533)
├─{_net1_qp0}(27534)
├─{_net1_qp1}(27535)
├─{_rng}(27526)
├─{http-server}(27504)
├─{seccomp_signal_}(27502)
├─{signal_handler}(27523)
├─{vcpu0}(27520)
├─{vcpu1}(27522)
└─{vmm}(27503)
Fixes: #2077
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The Windows image is quite large (about 20GiB), hence it takes some time
to copy it for every test in order to avoid potential corruption.
One way to mitigate that without compromising on safety between each
test is by using device mapper. By creating a read-only base, we ensure
the image won't be modified by any of the tests, and by creating one
snapshot for each test, we avoid copying the entire image each time.
A dedicated Copy On Write disk image is created to handle any change
that might be performed on the base image, letting the tests behave as
expected.
Fixes#2155
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By relying on the Guest object, Windows dedicated tests copy the Windows
guest image before booting from it. The point being to avoid corruption
between multiple tests. This is already how the rest of the integration
tests work, Windows tests were the only ones missing this feature.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The tight coupling of these packages caused issues for dependabot trying
to update them separately so instead combine them together into one
update.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This image does not have the pollinate service which can sometimes fail
and prevent SSH from starting as it marks itself as a prerequisite. This
service will never fully succeed as it tries to make a network
connection which will fail inside our test VMs.
Fixes: #2113
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This allows the removal of most of the custom patches in particular all
the virtio-fs ones have been integrated.
Fixes: #2070
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The ch-5.10.6 branch is rebased on top of Linux stable and includes the
latest virtio-iommu support patches.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Using --net=host is not necessary for any of the integration tests, so
let's use the default network option called "bridge".
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Older libc (like RHEL7) uses open() rather than openat(). This was
demonstrated through a failure to open /etc/localtime as used by
gmtime() libc call trigged from the vCPU thread (CMOS device.)
Fixes: #2111
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Because we're back on transient builder, let's download the image
everytime.
This reverts commit b5653d5278.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Some sporadic failures were due to an early connection to the VM while
it was not fully ready. Increasing sleep times fixes these issues.
Fixes#2104
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Sometimes when running under the CI tests fail due to a barrier not
being released and the guest blocks on an MMIO write. Add further
debugging to try and identify the issue.
See: #2118
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Given we already check the connected IP address matches the expected
guest IP address, the check on the "booted" message is not needed.
Fixes: #2117
Signed-off-by: Bo Chen <chen.bo@intel.com>
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
On aarch64, the openat() syscall was missing from the seccomp filters
list, preventing the test_watchdog from running properly.
Fixes#2103
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Temporarily disable fast failing to try and make progress on CI
stabilisation to allow more test coverage.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This test is very flaky and regularly causing CI failures. Until we can
identify the root cause we should disable this test.
See: #2103
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Simplify our image handling by not copying both QCOW2 and raw images for
every test. Allow the test to choose QCOW2 or raw by specifying the
image name manually. A follow on patch will add explicity QCOW2 tests.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When an SSH command fails we want to be able to see, via a panic() why
and where it failed. Replace use of .unwrap_or_default() from SSH
command calls to ensure that we can see the location of the panic.
Also enhance the existing SSH output code to show the error if there is
one.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Based on the LKML feedback, the devices under /dev/sgx/* are
not justified. SGX RFC v40 moves the SGX device nodes to /dev/sgx_*
and this is reflected in kvm-sgx (next branch) too.
Update cloud-hypervisor code and documentation to follow this.
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
This reverts commit 7f1f335950.
This breaks the compilation on Fedora/RedHat machines by requiring
dpkg-deb in the build.rs
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This allows the removal of most of the custom patches in particular all
the virtio-fs ones have been integrated.
Partially fixes: #2070
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
If the vCPU thread calls log!() the time difference between the call
time and the boot up time is reported. On most environments and
architectures this covered by a vDSO call rather than a syscall. However
on some platforms this turns into a syscall.
Fixes: #2080
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
error: field assignment outside of initializer for an instance created with Default::default()
Error: --> hypervisor/src/kvm/mod.rs:1239:9
|
1239 | state.mp_state = self.get_mp_state()?;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= note: `-D clippy::field-reassign-with-default` implied by `-D warnings`
note: consider initializing the variable with `kvm::aarch64::VcpuKvmState { mp_state: self.get_mp_state()?, ..Default::default() }` and removing relevant reassignments
--> hypervisor/src/kvm/mod.rs:1237:9
|
1237 | let mut state = CpuState::default();
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#field_reassign_with_default
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
error: field assignment outside of initializer for an instance created with Default::default()
--> virtio-devices/src/mem.rs:496:9
|
496 | resp.resp_type = resp_type;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
note: consider initializing the variable with `mem::VirtioMemResp { resp_type: resp_type, ..Default::default() }` and removing relevant reassignments
--> virtio-devices/src/mem.rs:495:9
|
495 | let mut resp = VirtioMemResp::default();
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#field_reassign_with_default
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
We must explicitly mark these values as u8 as the function that consumes
them takes a T and needs to use the specific width.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
error: field assignment outside of initializer for an instance created with Default::default()
--> hypervisor/src/kvm/mod.rs:318:9
|
318 | cap.cap = KVM_CAP_SPLIT_IRQCHIP;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= note: `-D clippy::field-reassign-with-default` implied by `-D warnings`
note: consider initializing the variable with `kvm_bindings::kvm_enable_cap { cap: KVM_CAP_SPLIT_IRQCHIP, ..Default::default() }` and removing relevant reassignments
--> hypervisor/src/kvm/mod.rs:317:9
|
317 | let mut cap: kvm_enable_cap = Default::default();
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#field_reassign_with_default
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
error: this returns a `Result<_, ()>
--> net_util/src/mac.rs:68:5
|
68 | pub fn from_bytes(src: &[u8]) -> Result<MacAddr, ()> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= note: `-D clippy::result-unit-err` implied by `-D warnings`
= help: use a custom Error type instead
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#result_unit_err
Replace with std::io::Error like other locations in the same file.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
With Rust 1.49 using attributes on a function parameter is not allowed.
The recommended workaround is to put it in a new block.
error[E0658]: attributes on expressions are experimental
--> vmm/src/memory_manager.rs:698:17
|
698 | #[cfg(target_arch = "x86_64")]
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= note: see issue #15701 <https://github.com/rust-lang/rust/issues/15701> for more information
error: removing an expression is not supported in this position
--> vmm/src/memory_manager.rs:698:17
|
698 | #[cfg(target_arch = "x86_64")]
|
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The boot time for direct kernel boot based tests is significantly
quicker than booting via the firmware and stock kernel as it triggers a
reboot during the boot process due to the initrd handling.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When doing a direct kernel boot only have console=ttyS0 in the command
line if we are explicitly testing the serial output. The default
behaviour is `--serial null` so this output will not be visible but will
trigger a KVM exit for every byte which is very costly when running
under nested virtualization.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
On the CI we are seeing issues with the activation barriers not being released:
cloud-hypervisor: 12.452434193s: INFO:vmm/src/vm.rs:413 -- Waiting for barrier
cloud-hypervisor: 12.452499794s: INFO:virtio-devices/src/block.rs:382 -- Changing cache mode to writeback
cloud-hypervisor: 12.452605195s: INFO:vmm/src/vm.rs:413 -- Waiting for barrier
cloud-hypervisor: 12.452684596s: INFO:virtio-devices/src/transport/pci_device.rs:671 -- Waiting for barrier
cloud-hypervisor: 12.452708196s: INFO:virtio-devices/src/transport/pci_device.rs:673 -- Barrier released
cloud-hypervisor: 12.452717596s: INFO:vmm/src/vm.rs:415 -- Barrier released
Add some debugging to try and identify the vause of this issue.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Now that we can support supplying an fd to the TAP device we can support
the MACVTAP and MACVLAN use cases.
Fixes: #872
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add an "fd=" parameter to allow specifying a TAP fd to use. Currently
only one fd for one queue pair is supported.
Fixes: #2052
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add support for creating virtio-net device from existing TAP fd.
Currently only a single fd and thus no-more than 2 queues (one pair) is
suppored.
Fixes: #2052
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Ultimately this will allow the creation of a virtio-net device that is
backed by a file descriptor.
This function ensures that the TAP device is correctly setup with
offloading and non-blocking.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
And fix related warnings: op_kind and op_register are being deprecated
as they might panic.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Starting the virtio device threads from the VMM thread has slowed down
the start of the VM when running on a highly contested system like the
CI.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When a device is ready to be activated signal to the VMM thread via an
EventFd that there is a device to be activated. When the VMM receives a
notification on the EventFd that there is a device to be activated
notify the device manager to attempt to activate any devices that have
not been activated.
As a side effect the VMM thread will create the virtio device threads.
Fixes: #1863
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
We need to be able to return the barrier from the code that prepares to
activate the virtio device. This triggered by a write to the
configuration fields stored in the PCI BAR. Since bars can be accessed
by both memory mapping and through PCI config I/O several prototypes
must be changed.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This can be uses to indicate to the caller that it should wait on the
barrier before returning as there is some asynchronous activity
triggered by the write which requires the KVM exit to block until it's
completed.
This is useful for having vCPU thread wait for the VMM thread to proceed
to activate the virtio devices.
See #1863
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Currently these two macros(msr, msr_data) reside both on kvm and mshv
module. Definition is same for both module. Moving them to arch/x86
module eliminates redundancy and makes more sense.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
On the CI we are seeing that sometimes the epoll is receiving these
errors which do not indicate a failure but that we should retry.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Since we can't test mshv and kvm at the same time, --all-features no
longer work.
We factorize all, non-hypervisor related features into a common set and
mix that with either mshv and kvm.
Co-Developed-by: Samuel Ortiz <sameo@linux.intel.com>
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Co-Developed-by: Wei Liu <liuwe@microsoft.com>
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Signed-off-by: Muminul Islam <muislam@microsoft.com>
We don't have IrqFd and IOEventFd support in the kernel for now.
So an emulation layer is needed. In the future, we will be adding this
support in the kernel.
Co-Developed-by: Wei Liu <liuwe@microsoft.com>
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Signed-off-by: Muminul Islam <muislam@microsoft.com>
vmmops trait object is needed to get access some
of the upper level vmm functionalities i.e guest
memory access, IO read write etc.
Co-Developed-by: Praveen Paladugu <prapal@microsoft.com>
Signed-off-by: Praveen Paladugu <prapal@microsoft.com>
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Adding hv_state (hyperv state) to Vm and Vcpu struct for mshv.
This state is needed to keep some kernel data(for now hypercall page)
in the vmm.
Co-Developed-by: Praveen Paladugu <prapal@microsoft.com>
Signed-off-by: Praveen Paladugu <prapal@microsoft.com>
Signed-off-by: Muminul Islam <muislam@microsoft.com>
This is the initial folder structure of the mshv module inside
the hypervisor crate. The aim of this module is to support Microsoft
Hyper-V as a supported Hypervisor.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
There are some code base and function which are purely KVM specific for
now and we don't have those supports in mshv at the moment but we have plan
for the future. We are doing a feature guard with KVM. For example, KVM has
mp_state, cpu clock support, which we don't have for mshv. In order to build
those code we are making the code base for KVM specific compilation.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
The customized hashmap macro can't be lifted to common MockVMM code.
MockVMM only needs a collection to iterate over to get initial register
states. A vector is just as good as a hashmap.
Switch to use a vector to store initial register states. This allows us
to drop the hashmap macro everywhere.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Unfortunately Rust stable does not yet have inline ASM support the flag
calculation will have to be implemented in software.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
nightly / beta are not required actions but if they fail then all other
builds will be cancelled and so prevent the PR from being cleared as
buildable.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Unfortunately it seems patch entries are ignored when obtaining
dependencies from another workspace.
Remove the problematic kvm-ioctls and kvm-bindings patch entries and use
the forked repository directly.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When using an PIO write to 0x80 which is a special case handle that and
then return without going through the resolve.
This removes an extra warning that is reported.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
There is no need to have a pair of curly brackets for structures without
any member.
No functional change.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
The mapping between code and its handler is static. We can drop the
HashMap in favour of a static match expression.
This has two benefits:
1. No more memory allocation and deallocation for the HashMap.
2. Shorter look-up time.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
When a total ordering between multiple atomic variables is not required
then use Ordering::Acquire with atomic loads and Ordering::Release with
atomic stores.
This will improve performance as this does not require a memory fence
on x86_64 which Ordering::SeqCst will use.
Add a comment to the code in the vCPU handling code where it operates on
multiple atomics to explain why Ordering::SeqCst is required.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The DeviceNode cannot be fully represented as it embeds a Rust style
enum (i.e. with data) which is instead represented by a simple
associative array.
Fixes: #1167
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When the x86 instruction decoder tells us about some missing bytes from
the instruction stream, we call into the platform fetch method and
emulate one last instruction.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
In preparation for the instruction fetching step, we modify the decoding
loop so that we can check what the last decoding error is.
We also switch to explictly using decode_out() which removes a 32 bytes
copy compared to decode().
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
In order to validate emulated memory accesses, we need to be able to get
all the segments descriptor attributes.
This is done by abstracting the SegmentRegister attributes through a
trait that each hypervisor will have to implement.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
We need to be able to build the CPU mode from its state in order to
start implementing mode related checks in the x86 emulator.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
If we rely on timeouts at the top level we can get builds being aborted
simply because they took too long to be scheduled rather than because
the actual integration tests took too long.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The MockVMM platform will be used by other instructions emulation
implementations, but also by the emulator framework.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The observation here is PlatformEmulator can be seen as the context for
emulation to take place. It should be rather easy to construct a context
that satisfies the lifetime constraints for instruction emulation.
The thread doing the emulation will have full ownership over the
context, so this removes the need to wrap PlatformEmulator in Arc and
Mutex, as well as the need for the context to be either Clone or Copy.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
The emulator gets a CPU state from a CpuStateManager instance, emulates
the passed instructions stream and returns the modified CPU state.
The emulator is a skeleton for now since it comes with an empty
instruction mnemonic map.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
And an InstructionMap helper structure to map x86 mnemonic codes
to instruction handlers.
Any instruction emulation implementation should then boil down with
implementing InstructionHandler for any supported mnemonic.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Minimal will be defined by the amount of emulated instructions.
Carrying all GPRs, all CRs, segment registers and table registers should
cover quite a few instructions.
Co-developed-by: Wei Liu <liuwe@microsoft.com>
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
For efficiently emulating x86 instructions, we need to build and pass a
CPU state copy/reference to instruction emulation handlers. Those handlers
will typically modify the CPU state and let the caller commit those
changes back through the PlatformEmulator trait set_cpu_state method.
Hypervisors typically have internal CPU state structures, that maps back
to the correspinding kernel APIs. By implementing the CpuState trait,
instruction emulators will be able to directly work on CPU state
instances that are directly consumable by the underlying hypervisor and
its kernel APIs.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
In order to emulate instructions, we need a way to get access to some of
the guest resources. The PlatformEmulator interface provides guest
memory and CPU state access to emulator implementations.
Typically, an hypervisor will implement PlatformEmulator for architecture
specific instruction emulators to build their framework on top of.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
We will need the GDT API for the hypervisor's x86 instruction
emulator implementation, it's better if the arch crate depends on the
hypervisor one rather than the other way around.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The configuration is stored separately to the Vm in the VMM. The failure
to store the config was preventing the VM from shutting down correctly
as Vmm::vm_delete() checks for the presence of the config.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Rather return the None to the caller to handle instead. This removes the
source of a potential panic.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The live migration support added use of this ioctl but it wasn't
included in the permitted list.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This interface is used by the vCPU thread to delegate responsibility for
handling MMIO/PIO operations and to support different approaches than a
VM exit.
During profiling I found that we were spending 13.75% of the boot CPU
uage acquiring access to the object holding the VmmOps via
ArcSwap::load_full()
13.75% 6.02% vcpu0 cloud-hypervisor [.] arc_swap::ArcSwapAny<T,S>::load_full
|
---arc_swap::ArcSwapAny<T,S>::load_full
|
--13.43%--<hypervisor::kvm::KvmVcpu as hypervisor::cpu::Vcpu>::run
std::sys_common::backtrace::__rust_begin_short_backtrace
core::ops::function::FnOnce::call_once{{vtable-shim}}
std::sys::unix::thread::Thread::new::thread_start
However since the object implementing VmmOps does not need to be mutable
and it is only used from the vCPU side we can change the ownership to
being a simple Arc<> that is passed in when calling create_vcpu().
This completely removes the above CPU usage from subsequent profiles.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
As we switched to focal for this test we no longer get any output during
the boot unless serial is used over virtio-console.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
There have been a lot of flakes around tests such as
test_virtio_fs_hotplug_dax_on_w_vhost_user_fs_daemon() or
test_virtio_fs_hotplug_dax_on() which all try and hotplug memory.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add config parameter to --disk called "_disable_io_uring" (the
underscore prefix indicating it is not for public consumpion.) Use this
option to disable io_uring if it would otherwise be used.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
With the removal of vhost-user self-spawning support we should migrate
the tests to use the binaries so that we can remove the functionality
from the cloud-hypervisor binary itself.
See: #1925
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Now the VM is paused/resumed by the migration process itself.
0. The guest configuration is sent to the destination
1. Dirty page log tracking is started by start_memory_dirty_log()
2. All guest memory is sent to the destination
3. Up to 5 attempts are made to send the dirty guest memory to the
destination...
4. ...before the VM is paused
5. One last set of dirty pages is sent to the destination
6. The guest is snapshotted and sent to the destination
7. When the migration is completed the destination unpauses the received
VM.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This allows code running in the VMM to access the VM's MemoryManager's
functionality for managing the dirty log including resetting it but also
generating a table.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Prior to sending the memory the full state is not needed only the
configuration. This is sufficient to create the appropriate structures
in the guest and have the memory allocations ready for filling.
Update the protocol documentation to add a separate config step and move
the state to after the memory is transferred. As the VM is created in a
separate step to restoring it the requires a slightly different
constructor as well as saving the VM object for the subsequent commands.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In order to do this we must extend the MemoryManager API to add the
ability to specify the tracking of the dirty pages when creating the
userspace mappings and also keep track of the userspace mappings that
have been created for RAM regions.
Currently the dirty pages are collected into ranges based on a block
level of 64 pages. The algorithm could be tweaked to create smaller
ranges but for now if any page in the block of 64 is dirty the whole
block is added to the range.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
While the addressable space size reduction of 4k in necessary due to
the Linux bug, the 64k alignment of the addressable space size is
required by Windows. This patch satisfies both.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
Cloning the ArcSwapOption (like the ArcSwap) does not act like a
.clone() on an Arc, instead an entirely new ArcSwap is created with the
same contents. To correctly share the ArcSwap needs to be placed inside
an Arc.
See: 2433d5719b (diff-6c6d94533c44c19bd1416ef17bad1a878e63dca6e98d59181228fbe8f967c62bR6)
Due to this being wrongly used ::clone() was removed from
ArcSwap/ArcSwapOption in 1.0.0.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This is tested by:
Source VMM:
target/debug/cloud-hypervisor --kernel ~/src/linux/vmlinux \
--pmem file=~/workloads/focal.raw --cpus boot=1 \
--memory size=2048M \
--cmdline"root=/dev/pmem0p1 console=ttyS0" --serial tty --console off \
--api-socket=/tmp/api1 -v
Destination VMM:
target/debug/cloud-hypervisor --api-socket=/tmp/api2 -v
And the following commands:
target/debug/ch-remote --api-socket=/tmp/api1 pause
target/debug/ch-remote --api-socket=/tmp/api2 receive-migration unix:/tmp/foo &
target/debug/ch-remote --api-socket=/tmp/api1 send-migration unix:/tmp/foo
target/debug/ch-remote --api-socket=/tmp/api2 resume
The VM is then responsive on the destination VMM.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This allows the code to be reused when creating the VM from a snapshot
when doing VM migration.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add API entry points with stub implementation for sending and receiving
a VM from one VMM to another.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Due to a known limitation in OpenAPITools/openapi-generator tool,
it's impossible to send go zero types, like false and 0 to
cloud-hypervisor because `omitempty` is added if a field is not
required.
Set cache_size, dax, num_queues and queue_size as required to remove
`omitempty` from the json tag.
fixes#1961
Signed-off-by: Julio Montes <julio.montes@intel.com>
This also removes the need to lookup up the "exe" symlink for finding
the VMM executable path.
Fixes: #1925
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The integration script failed to obtain the new guest kernel commit when
the code folder had been existing on CI machine.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
The logic to handle AArch64 system event was: SHUTDOWN and RESET were
all treated as RESET.
Now we handle them differently:
- RESET event will trigger Vmm::vm_reboot(),
- SHUTDOWN event will trigger Vmm::vm_shutdown().
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Now Vcpu::run() returns a boolean value to VcpuManager, indicating
whether the VM is going to reboot (false) or just continue (true).
Moving the handling of hypervisor VCPU run result from Vcpu to
VcpuManager gives us the flexibility to handle more scenarios like
shutting down on AArch64.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Set the test case test_snapshot_restore X86 only, instead of excluding
it from test command line.
The command line option was added because we used to support migration
with Virtio-MMIO, but not Virtio-PCI.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Rather than filling the guest memory from a file at the point of the the
guest memory region being created instead fill from the file later. This
simplifies the region creation code but also adds flexibility for
sourcing the guest memory from a source other than an on disk file.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Tests not ported include 1) the ones that start guest VMs without
network (e.g. test_net_hotplug, test_initramfs), 2) test_vfio that
involves l2 guest. Also, some tests that use bionic guest image are
given extended timeout (120s) for 'wait_vm_boot'.
Signed-off-by: Bo Chen <chen.bo@intel.com>
Instead of waiting blindly with fixed amount of sleeping time, we can
use the `wait-timeout` crate to explicitly wait VM shutdown (with a
timeout). It can reduces the execution time of some tests
substantially. Also, this patch increases the `shutdown` timeout for
'test_reboot', which should fix the recent sporadic failures on this
test.
Signed-off-by: Bo Chen <chen.bo@intel.com>
Address build failure from activity in the development virtio-fs branch
by using the stable fork.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Instead of blindly waiting for 20-40s for the guest VM to boot, this
patch waits the notification from the guest VM explicitly by using a
simple TcpListener on the host and a custom systemd service in the
guest.
This patch also ported few tests to use this new machanism, while more
tests are to be ported.
Signed-off-by: Bo Chen <chen.bo@intel.com>
As a mirror of bdbea19e23 which ensured
that GuestMemoryMmap::read_exact_from() was used to read all the file to
the region ensure that all the guest memory region is written to disk.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Use a Result<> type with an error to simplify the code in start_vmm().
This will also make it easier to add cleanup funtionality.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This gives a nicer user experience and this error can now be used as the
source for other errors based off this.
See: #1910
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Now that we that our CI is running with a kernel that is new enough to
support io_uring we can turn this feature on by default.
See: #1561
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Any occurrence of of a variable containing `ext_region` is replaced with
the less confusing name `saved_region`. The point is to clearly identify
the memory regions that might have been saved during a snapshot, while
the `ext` standing for `external` was pretty unclear.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In the context of saving the memory regions content through snapshot,
using the term "backing file" brings confusion with the actual backing
file that might back the memory mapping.
To avoid such conflicting naming, the 'backing_file' field from the
MemoryRegion structure gets replaced with 'content', as this is
designating the potential file containing the memory region data.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Use GuestRegionMmap::read_exact_from() to ensure that all of the file is
read into the guest. This addresses an issue where
GuestRegionMmap::read_from() was only copying the first 2GiB of the
memory and so lead to snapshot-restore was failing when the guest RAM
was 2GiB or greater.
This change also propagates any error from the copying upwards.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Split out the HTTP request handling code from ch-remote into a new
crate which can be used in other places where talking to the API server
by HTTP is necessary.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Use the trait for Read/Write rather than specifying the concrete type.
This allows for the functionality to be used for different socket types.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When restoring if a region of RAM is backed by anonymous memory i.e from
memfd_create() then copy the contents of the ram from the file that has
been saved to disk.
Previously the code would map the memory from that file into the guest
using a MAP_PRIVATE mapping. This has the effect of
minimising the restore time but provides an issue where the restored VM
does not have the same structure as the snapshotted VM, in particular
memory is backed by files in the restored VM that were anonymously
backed in the original.
This creates two problems:
* The snapshot data is mapped from files for the pages of the guest
which prevents the storage from being reclaimed.
* When snapshotting again the guest memory will not be correctly saved
as it will have looked like it was backed by a file so it will not be
written to disk but as it is a MAP_PRIVATE mapping the changes will
never be written to the disk again. This results in incorrect
behaviour.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Now that virtio-balloon is not declared as part of the --memory
parameter, the integration tests are updated to keep the correct
behavior.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The standalone `--balloon` parameter being fully functional at this
point, we can get rid of the balloon options from the --memory
parameter.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that we have a new dedicated way of asking for a balloon through the
CLI and the REST API, we can move all the balloon code to the device
manager. This allows us to simplify the memory manager, which is already
quite complex.
It also simplifies the behavior of the balloon resizing command. Instead
of providing the expected size for the RAM, which is complex when memory
zones are involved, it now expects the balloon size. This is a much more
straightforward behavior as it really resizes the balloon to the desired
size. Additionally to the simplication, the benefit of this approach is
that it does not need to be tied to the memory manager at all.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This introduces a new way of defining the virtio-balloon device. Instead
of going through the --memory parameter, the idea is to consider balloon
as a standalone virtio device.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The snasphot/restore feature is not working because some CPU states are
not properly saved, which means they can't be restored later on.
First thing, we ensure the CPUID is stored so that it can be properly
restored later. The code is simplified and pushed down to the hypervisor
crate.
Second thing, we identify for each vCPU if the Hyper-V SynIC device is
emulated or not. In case it is, that means some specific MSRs will be
set by the guest. These MSRs must be saved in order to properly restore
the VM.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The watchdog device is created through the "--watchdog" parameter. At
most a single watchdog can be created per VM.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This device operates a single virtq. When the driver offers a descriptor
to the device it is interpreted as a "ping" to indicate that the guest
is alive. A periodic timer fires and if when the timer is fired there
has not been a "ping" from the guest then the device will reset the VM.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Before Virtio-mmio was removed, we passed an optional PCI space address
parameter to AArch64 code for generating FDT. The address is none if the
transport is MMIO.
Now Virtio-PCI is the only option, the parameter is mandatory.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Given the increased amount of output from cloud-hypervisor, this patch
also increased the PIPE_SIZE to 32MB (from 256KB).
Signed-off-by: Bo Chen <chen.bo@intel.com>
This patch prints the complete commandline when launching
cloud-hypervisor. It also prints the details of the `ssh` command if
the command is failing.
Signed-off-by: Bo Chen <chen.bo@intel.com>
Virtio-mmio is removed, now virtio-pci is the only option for virtio
transport layer. We use MSI for PCI device interrupt. While GICv2, the
legacy interrupt controller, doesn't support MSI. So GICv2 is not very
practical for Cloud-hypervisor, we can remove it.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
When shutting down a VM using VFIO, the following error has been
detected:
vfio-ioctls/src/vfio_device.rs:312 -- Could not delete VFIO group:
KvmSetDeviceAttr(Error(9))
After some investigation, it appears the KVM device file descriptor used
for removing a VFIO group was already closed. This is coming from the
Rust sequence of Drop, from the DeviceManager all the way down to
VfioDevice.
Because the DeviceManager owns passthrough_device, which is effectively
a KVM device file descriptor, when the DeviceManager is dropped, the
passthrough_device follows, with the effect of closing the KVM device
file descriptor. Problem is, VfioDevice has not been dropped yet and it
still needs a valid KVM device file descriptor.
That's why the simple way to fix this issue coming from Rust dropping
all resources is to make Linux accountable for it by duplicating the
file descriptor. This way, even when the passthrough_device is dropped,
the KVM file descriptor is closed, but a duplicated instance is still
valid and owned by the VfioContainer.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We turn on that emulation for Windows. Windows does not have KVM's PV
clock, so calling notify_guest_clock_paused results in an error.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
While using the virtio-iommu device involving L2 scenario, and tearing
things down all the way from L2 back to L0 exposed some bad syscalls
that were not part of the authorized list.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that we've written Windows integration tests and the associated
script to launch them, this patch enables the support for Windows tests
in dev_cli.sh, so that we can run it in our Cloud Hypervisor container.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This is a new integration test running Windows as a guest with Cloud
Hypervisor. Once the VM is booted, the test connects to the guest
through SSH and shutdown the VM. If this succeeds, this means the VM
was properly booted to userspace and that the network was functional.
Important to note that because this test generates lots of logs, it
requires a large pipe size for both stdout and stderr.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Introduce a new test that will validate the new option `max_phys_bits`
from the `--cpus` parameter behaves as expected.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
If the user specified a maximum physical bits value through the
`max_phys_bits` option from `--cpus` parameter, the guest CPUID
will be patched accordingly to ensure the guest will find the
right amount of physical bits.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
If the user provided a maximum physical bits value for the vCPUs, the
memory manager will adapt the guest physical address space accordingly
so that devices are not placed further than the specified value.
It's important to note that if the number exceed what is available on
the host, the smaller number will be picked.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to let the user choose maximum address space size, this patch
introduces a new option `max_phys_bits` to the `--cpus` parameter.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The 'GuestAddress::unchecked_add' function has undefined behavior when
an overflow occurs. Its alternative 'checked_add' requires use to handle
the overflow explicitly.
Signed-off-by: Bo Chen <chen.bo@intel.com>
We are now reserving a 256M gap in the guest address space each time
when hotplugging memory with ACPI, which prevents users from hotplugging
memory to the maximum size they requested. We confirm that there is no
need to reserve this gap.
This patch removes the 'reserved gaps'. It also refactors the
'MemoryManager::start_addr' so that it is rounding-up to 128M alignment
when hotplugged memory is allowed with ACPI.
Signed-off-by: Bo Chen <chen.bo@intel.com>
In previous dev_cli.sh, the `uname -m` command will generate
either `x86_64` or `aarch64`, which is inconsistent with the
architectures in the Dockerfile, namely `amd64` and `arm64`.
This will cause some dependancy missing in the docker container
when the docker image is built locally.
This commit fixes this inconsistancy.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
We now try to create a ram region of size 0 when the requested memory
size is the same as current memory size. It results in an error of
`GuestMemoryRegion(Mmap(Os { code: 22, kind: InvalidInput, message:
"Invalid argument" }))`. This error is not meaningful to users and we
should not report it.
Signed-off-by: Bo Chen <chen.bo@intel.com>
A failure appeared in AArch64 musl cross build, after upgrading rust
to v1.47.0. A symbol "strrchr" was missing while linking against
static libfdt.a.
The issue could be caused by missing symbol(s) in new rust toolchain.
This fix pins the rust version in this cross build action to a stable-
enough version. Further upgrade will be done manually after testing.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
"Using a stable sort consumes more memory and cpu cycles. Because values
which compare equal are identical, preserving their relative order (the
guarantee that a stable sort provides) means nothing, while the extra
costs still apply."
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This is a new clippy check introduced in 1.47 which requires the use of
the matches!() macro for simple match blocks that return a boolean.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Instead of having the hypervisor crate embedding Cloud-Hypervisor forks
from the rust-vmm project, it's more appropriate to leave the rust-vmm
references in the hypervisor crate, and have the root Cargo.toml being
patched.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This changeset extends the documentation with the UEFI and Windows
related info. The focus is on providing consumer with a minimum
necessary and proper piece of the information to enter the features
quickly. While UEFI is a cross platform topic, it is a required
prerequisite for the Windows usage.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
The OneRegister literally means "one (arbitrary) register". Just call it
"Register" instead. There is no need to inherit KVM's naming scheme in
the hypervisor agnostic code.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
The goal here is to replace anywhere possible a virtio structure
with a "C, packed" representation by a "C" representation. Some
virtio structures are not expected to be packed, therefore there's
no reason for using the more restrictive "C, packed" representation.
This is important since "packed" representation can still cause
undefined behaviors with Rust 2018.
By removing the need for "packed" representation, we can simplify a
bit of code by deriving the Serialize trait without writing the
implementation ourselves.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Small patch creating a dedicated `block_io_uring_is_supported()`
function for the non-io_uring case, so that we can simplify the
code in the DeviceManager.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The purpose of that step is to make sure each commit builds. The `check`
command is much faster for that purpose.
On my 32-core machine `cargo check --all` takes around 25 seconds while
`cargo build --all` takes around 35 seconds, so that's quite a bit of
time saving there.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Since all unit and integration tests are run inside containers because
they are called from dev_cli.sh, they always run as root. That's why
both unit and integration scripts can be simplified as they don't need
to apply specific capabilities and run cargo tests in a dedicated 'kvm'
group.
Fixes#1683
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Windows will write zero to the IOAPIC version register as part of
probing if the device is there.
Fixes: #1791
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The existing virtio-blk hotplug test is extended by removing and
re-adding the virtio-blk device. This ensures the unplug/re-plug
feature is properly tested.
Fixes#1809
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The existing virtio-net hotplug test is extended by removing and
re-adding the virtio-net device. This ensures the unplug/re-plug
feature is properly tested.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Without the unlink(2) syscall being allowed, Cloud-Hypervisor crashes
when we remove a virtio-vsock device that has been previously added.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
It's important to ensure Cloud-Hypervisor's Git tree is bisectable, and
that's why this commit extends the existing build.yaml Github action.
This will validate that each commit from an incoming pull request can be
built.
Fixes#1808
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The Cloud-Hypervisor fork of the vhost crate contains one small
additional patch compared to the rust-vmm upstream version, meant for
increasing the connection timeout.
This patch is intended to be merged in order to check if it helps our CI
fixing the vhost-user-blk flakes that we've been observing recently.
If it fixes it, we'll submit a similar patch upstream and switch back to
the upstream vhost crate, otherwise we'll simply switch back to the
upstream crate, discarding this patch.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Because of the PCI refactoring that happened in the previous commit
d793cc4da3, the ability to fully remove a
PCI device was altered.
The refactoring was correct, but the usage of a generic function to pass
the same reference for both BusDevice, PciDevice and Any + Send + Sync
causes the Arc::ptr_eq() function to behave differently than expected,
as it does not match the references later in the code. That means we
were not able to remove the device reference from the MMIO and/or PIO
buses, which was leading to some bus range overlapping error once we
were trying to add a device again to the previous range that should have
been removed.
Fixes#1802
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Run loop in hypervisor needs a callback mechanism to access resources
like guest memory, mmio, pio etc.
VmmOps trait is introduced here, which is implemented by vmm module.
While handling vcpuexits in run loop, this trait allows hypervisor
module access to the above mentioned resources via callbacks.
Signed-off-by: Praveen Paladugu <prapal@microsoft.com>
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
A new version of vm-memory was released upstream which resulted in some
components pulling in that new version. Update the version number used
to point to the latest version but continue to use our patched version
due to the fix for #1258
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The PMEM support has an option called "discard_writes" which when true
will prevent changes to the device from hitting the backing file. This
is trying to be the equivalent of "readonly" support of the block
device.
Previously the memory of the device was marked as KVM_READONLY. This
resulted in a trap when the guest attempted to write to it resulting a
VM exit (and recently a warning). This has a very detrimental effect on
the performance so instead make "discard_writes" truly CoW by mapping
the memory as `PROT_READ | PROT_WRITE` and using `MAP_PRIVATE` to
establish the CoW mapping.
Fixes: #1795
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When all refblocks are consumed, the loop looking for the first free
cluster would access the element at refcounts[refcounts.len()], which is
out of bounds. Modify the free cluster search loop to check that the
index is in bounds before accessing it.
Tested-by: Daniel Verkamp <dverkamp@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Commit-Queue: Daniel Verkamp <dverkamp@chromium.org>
(cherry picked from crosvm commit f21572c7187c8beb9c6bfea6446351ae93200d01)
Fixes: #1792
Signed-off-by: Bo Chen <chen.bo@intel.com>
The virtio-balloon change the memory size is asynchronous.
VirtioBalloonConfig.actual of balloon device show current balloon size.
This commit add memory_actual_size to vm.info to show memory actual size.
Signed-off-by: Hui Zhu <teawater@antfin.com>
When the destination mode is physical, the destination field should
only be defined through bits 56-59, as defined in the IOAPIC spec. But
from the APIC specification, the APIC ID is always defined on 8 bits no
matter which destination mode is selected. That's why we always retrieve
the destination field based on bits 56-63.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This patch adds two required dependencies to fuzz/Cargo.toml, and fixes
the building error on the 'block' fuzzer.
Signed-off-by: Bo Chen <chen.bo@intel.com>
Now that Docker images are automatically generated for both amd64 and
arm64 architectures, there's no need to generate the arm64 image locally
on the ARM CI during a CI run. The image should be available from
DockerHub instead.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Write to the exit_evt EventFD which will trigger all the devices and
vCPUs to exit. This is slightly cleaner than just exiting the process as
any temporary files will be removed.
Fixes: #1242
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This commit moves back to the branch "virtio-fs-dev" from virtiofsd, as
we figured the changes needed to use this branch and the requirements
from the new meson build from QEMU.
It updates the container version to ensure the dev_cli.sh script will
rely on the latest container which contains the needed packages.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By fixing the Dockerfile, we have now finalized the automated generation
of the Docker images for both architectures (amd64 and arm64).
Fixes#953
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This patch adds the missing the `iommu` and `id` option for
`VmAddDevice` in the openApi yaml to respect the internal data structure
in the code base. Also, setting the `id` explicitly for VFIO device
hotplug is required for VFIO device unplug through openAPI calls.
Signed-off-by: Bo Chen <chen.bo@intel.com>
According to openAPI specification [1], the format for `integer` types
can be only `int32` or `int64`, unsigned and 8-bits integers are not
supported.
This patch replaces `uint64` with `int64`, `uint32` with `int32` and
`uint8` with `int32`.
[1]: https://swagger.io/specification/#data-types
Signed-off-by: Julio Montes <julio.montes@intel.com>
MsiInterruptGroup doesn't need to know the internal field names of
InterruptRoute. Introduce two helper functions to eliminate references
to irq_fd. This is done similarly to the enable and disable helper
functions.
Also drop the pub keyword from InterruptRoute fields. It is not needed
anymore.
No functional change.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
In order to support both amd64 and arm64, we rely on the TARGETARCH
variable that is passed from the docker buildx command, based on the
platform used to build the container image.
There is no way to rely directly on $(uname -m) to assign a variable
with the correct x86_64 or aarch64 values we're looking for. Both ENV
and ARG don't evaluate the command, which means they see it as a simple
string. Using RUN is the only way to evaluate a command.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to build virtiofsd from the latest build system, the Python
package python3-setuptools is required.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The idea is to rely on this new Github Action to detect when the
Dockerfile is updated after a push to the master branch on the
repository.
Once triggered, this action builds the Docker image for both
linux/amd64 and linux/arm64 platforms, and updates it directly
on Docker Hub.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
According to openAPI specification[1], the format for `integer` types
can be only `int32` or `int64`, unsigned integers are not supported.
This patch replaces `uint64` with `int64`.
[1]: https://swagger.io/specification/#data-types
Signed-off-by: Julio Montes <julio.montes@intel.com>
In order to speed up the Linux boot (so as to avoid it having to scan a
large number of pages) place the MP table directly after the SMBIOS
table if there is sufficient room. The start address of the SMBIOS table
is one of the three (and the largest) location that the MP table can
also be located at.
Before:
[ 0.000399] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT
[ 0.014945] check: Scanning 1 areas for low memory corruption
After:
[ 0.000284] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT
[ 0.000421] found SMP MP-table at [mem 0x000f0090-0x000f009f]
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
There is no point in manually dropping the lock for gsi_msi_routes then
instantly grabbing it again in set_gsi_routes.
Make set_gsi_routes take a reference to the routing hashmap instead.
No functional change intended.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
The MTRR feature was missing from the CPUID, which is causing the guest
to ignore the MTRR settings exposed through dedicated MSRs.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Removing the ISA DMA configurations prevents the kernel from accessing
the port I/O 0x87, which was generating the following warning:
WARN:vmm/src/cpu.rs:378 -- Guest PIO read to unregistered address 0x87
Removing the TELCLOCK configuration prevents the kernel from accessing
the port I/O reserved for the memory manager, which was causing the
following warning:
WARN:vmm/src/memory_manager.rs:289 -- Unexpected offset for accessing
memory manager device: 15
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The ::new() does very little beyond trying to open the /dev/kvm device
so provide a hint to the user about what has gone wrong.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
As discussed in #1707, the `vcpu` thread can be stalled when using
`--serial tty`. To workaround that issue, this patch enforces to resize
the pipe size to 256K when we capture the stdout/stderr of the
cloud-hypervisor child process in the integration tests. Note that the
pipe size (256K) is chosen based on the output size of our integration
tests at this point, which may need to be increased in the future.
Signed-off-by: Bo Chen <chen.bo@intel.com>
By looking at Linux kernel boot time, we identified that a lot of time
was spent registering and unregistering IRQ fds to KVM. This is not
efficient and certainly not a wrong behavior from the Linux kernel,
but rather a problem with the Cloud-Hypervisor's implementation of
MSI-X.
The way to fix this issue is by ensuring the initial conditions are
correct, which means the entire MSI-X vector table must be disabled
and masked. Additionally, each vector must be individually masked.
With these correct conditions, Linux won't start masking interrupt
vectors, and later unmask them since they will be seen as masked from
the beginning. This means the OS will simply have to unmask them when
needed, avoiding the extra operation.
Another aspect of this patch is to prevent Cloud-Hypervisor from
enabling (by registering IRQ fd) all vectors when either the global
'mask' or 'enable' bits are set. Instead, we can simply let the mask()
and unmask() operations take care of it if needed.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Since Cloud-Hypervisor currently support one single PCI bus, we must
reflect this through the MCFG table, as it advertises the first bus and
the last bus available. In this case both are bus 0.
This patch saves quite some time during guest kernel boot, as it
prevents from checking each bus for available devices.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
When created, the IOAPIC entries should be masked, as it is the guest's
responsibility (FW and/or OS) to unmask them if/when necessary.
This patch saves a full round of port I/O writes from the guest to the
IOAPIC, meant for masking the unmasked entries.
Because they're now masked, the entries are not enabled, which means
they are not connected from a KVM perspective, saving from unneeded
registration/unregistration of the irq fds.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
While we figure out the details on how to correctly build virtiofsd from
the latest rebase from the branch "virtio-fs-dev" (which now relies on
QEMU's new build system), let's fix the CI by relying on an older branch
which still relies on the previous build system.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The states of GIC should be part of the VM states. This commit
enables the AArch64 VM states save/restore by adding save/restore
of GIC states.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Currently for AArch64, the GICv3-ITS is tried to be created first
when PCI is not needed, which is unnecessary. This commit fixes
the problem.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
The definition of libc::SYS_ftruncate on AArch64 is different
from that on x86_64. This commit unifies the previously hard-coded
syscall number for AArch64.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
`KVM_GET_REG_LIST` ioctl is needed in save/restore AArch64 vCPU.
Therefore we whitelist this ioctl in seccomp.
Also this commit unifies the `SYS_FTRUNCATE` syscall for x86_64
and AArch64.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Similarly as the VM booting process, on AArch64 systems,
the vCPUs should be created before the creation of GIC. This
commit refactors the vCPU save/restore code to achieve the
above-mentioned restoring order.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Since calling `KVM_GET_ONE_REG` before `KVM_VCPU_INIT` will
result in an error: Exec format error (os error 8). This commit
decouples the vCPU init process from `configure_vcpus`. Therefore
in the process of restoring the vCPUs, these vCPUs can be
initialized separately before started.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
The construction of `GICR_TYPER` register will need vCPU states.
Therefore this commit adds methods to extract saved vCPU states
from the cpu manager.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Unlike x86_64, the "interrupt_controller" in the device manager
for AArch64 is only a `Gic` object that implements the
`InterruptController` to provide the interrupt delivery service.
This is not the real GIC device so that we do not need to save
its states. Also, we do not need to insert it to the device_tree.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
The value of GIC register `GICR_TYPER` is needed in restoring
the GIC states. This commit adds a field in the GIC device struct
and a method to construct its value.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
In AArch64 systems, the state of GIC device can only be
retrieved from `KVM_GET_DEVICE_ATTR` ioctl. Therefore to implement
saving/restoring the GIC states, we need to make sure that the
GIC object (either the file descriptor or the device itself) can
be extracted after the VM is started.
This commit refactors the code of GIC creation by adding a new
field `gic_device_entity` in device manager and methods to set/get
this field. The GIC object can be therefore saved in the device
manager after calling `arch::configure_system`.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This commit adds a function which allows to save RDIST pending
tables to the guest RAM, as well as unit test case for it.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This commit adds the unit test cases for getting/setting the GIC
distributor, redistributor and ICC registers.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This commit implements the `get_device_attr` method for the
`KVM_GET_DEVICE_ATTR` ioctl. This ioctl will be used in retrieving
the GIC status.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This commit moves the GIC-related code to a separate module.
Therefore the implementation of GIC registers can be introduced
to the new module.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Adds 3 more unit test cases for AArch64:
*save_restore_core_regs
*save_restore_system_regs
*get_set_mpstate
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This commit adds methods to save/restore AArch64 vCPU registers,
including:
1. The AArch64 `VcpuKvmState` structure.
2. Some `Vcpu` trait methods of the `KvmVcpu` structure to
enable the save/restore of the AArch64 vCPU states.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This commit ports code from firecracker and refactors the existing
AArch64 code as the preparation for implementing save/restore
AArch64 vCPU, including:
1. Modification of `arm64_core_reg` macro to retrive the index of
arm64 core register and implemention of a helper to determine if
a register is a system register.
2. Move some macros and helpers in `arch` crate to the `hypervisor`
crate.
3. Added related unit tests for above functions and macros.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
virtio-mem device would use 'VIRTIO_MEM_F_ACPI_PXM' to add memory to NUMA
node, which MUST be existed, otherwise it will be assigned to node id 0,
even if user specify different node id.
According ACPI spec about Memory Affinity Structure, system hardware
supports hot-add memory region using 'Hot Pluggable | Enabled' flags.
Signed-off-by: Jiangbo Wu <jiangbo.wu@intel.com>
Use zone.host_numa_node to create memory zone, so that memory zone
can apply memory policy in according with host numa node ID
Signed-off-by: Jiangbo Wu <jiangbo.wu@intel.com>
If after the creation of the self-spawned backend, the VMM cannot create
the corresponding vhost-user frontend, the VMM must kill the freshly
spawned process in order to ensure the error propagation can happen.
In case the child process would still be around, the VMM cannot return
the error as it waits onto the child to terminate.
This should help us identify when self-spawned failures are caused by a
connection being refused between the VMM and the backend.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
When the VMM is terminated by receiving a SIGTERM signal, the signal
handler thread must be able to invoke ioctl(TCGETS) and ioctl(TCSETS)
without error.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Build testing of changes happens on GitHub actions and the integration
tests will build the binary (with different feature flags) again. So
these earlier build operations are just wasted time on the critical
path.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Only use `cloud-hypervisor` when referring to the binary itself and
prefer Cloud Hypervisor when referring to the project.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Some operations complete directly after they have been submitted, which
means they are not submitted asynchronously and therefore they don't
generate any ioevent. This is the reason why we are not processing some
of the completed operations, which leads to some unpredictable
behaviors.
Forcing all io_uring operations submitted to the SQE to be asynchronous
helps simplifying the code as it ensures the completion of every
operation will generate an ioevent, therefore no operation is missed.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that virtio-mem supports reboot, we extend the existing integration
tests to validate the amount of RAM after reboot is the same as before
the reboot, but also that we can still resize down the VM or the memory
zone after the reboot.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The missing syscall rt_sigprocmask(2) was triggered for the musl build
upon rebooting the VM, and was causing the VM to be killed.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to facilitate the memory hotplug for our users, let's enable
the automatic memory onlining in our guest kernel by activating the
kernel config option 'CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE'.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on all the preparatory work achieved through previous commits,
this patch updates the 'hotplugged_size' field for both MemoryConfig and
MemoryZoneConfig structures when either the whole memory is resized, or
simply when a memory zone is resized.
This fixes the lack of support for rebooting a VM with the right amount
of memory plugged in.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Adding a new field to VirtioMemZone structure, as it lets us associate
with a particular virtio-mem region the amount of memory that should be
plugged in at boot.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This patch simplifies the code as we have one single Option for the
VirtioMemZone. This also prepares for storing additional information
related to the virtio-mem region.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Add the new option 'hotplugged_size' to both --memory-zone and --memory
parameters so that we can let the user specify a certain amount of
memory being plugged at boot.
This is also part of making sure we can store the virtio-mem size over a
reboot of the VM.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit gives the possibility to create a virtio-mem device with
some memory already plugged into it. This is preliminary work to be
able to reboot a VM with the virtio-mem region being already resized.
Signed-off-by: Hui Zhu <teawater@antfin.com>
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that e820 tables are created from the 'boot_guest_memory', we can
simplify the memory manager code by adding the virtio-mem regions when
they are created. There's no need to wait for the first hotplug to
insert these regions.
This also anticipates the need for starting a VM with some memory
already plugged into the virtio-mem region.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to differentiate the 'boot' memory regions from the virtio-mem
regions, we store what we call 'boot_guest_memory'. This is useful to
provide the adequate list of regions to the configure_system() function
as it expects only the list of regions that should be exposed through
the e820 table.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Extend the existing test to validate that each NUMA node gets assigned
the right amount of memory after each memory zone has been resized.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that we can resize each memory zone independently, this commit
extends the memory zone related test by validating 'vm.resize-zone'
works correctly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The virtio-mem driver is generating some warnings regarding both size
and alignment of the virtio-mem region if not based on 128MiB:
The alignment of the physical start address can make some memory
unusable.
The alignment of the physical end address can make some memory
unusable.
For these reasons, the current patch enforces virtio-mem regions to be
128MiB aligned and checks the size provided by the user is a multiple of
128MiB.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that virtio-mem device accept a guest NUMA node as parameter, we
retrieve this information from the list of NUMA nodes. Based on the
memory zone associated with the virtio-mem device, we obtain the NUMA
node identifier, which we provide to the virtio-mem device.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Implement support for associating a virtio-mem device with a specific
guest NUMA node, based on the ACPI proximity domain identifier.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
For more consistency and help reading the code better, this commit
renames all 'virtiomem*' variables into 'virtio_mem*'.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By testing manually the memory resizing through virtio-mem, several
missing syscalls have been identified.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Implement a new VM action called 'resize-zone' allowing the user to
resize one specific memory zone at a time. This relies on all the
preliminary work from the previous commits to resize each virtio-mem
device independently from each others.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By adding a new parameter 'id' to the virtiomem_resize() function, we
prepare this function to be usable for both global memory resizing and
memory zone resizing.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
It's important to return the region covered by virtio-mem the first time
it is inserted as the device manager must update all devices with this
information.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the previous code changes, we can now update the MemoryManager
code to create one virtio-mem region and resizing handler per memory
zone. This will naturally create one virtio-mem device per memory zone
from the DeviceManager's code which has been previously updated as well.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In anticipation for resizing support of an individual memory zone,
this commit introduces a new option 'hotplug_size' to '--memory-zone'
parameter. This defines the amount of memory that can be added through
each specific memory zone.
Because memory zone resize is tied to virtio-mem, make sure the user
selects 'virtio-mem' hotplug method, otherwise return an error.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Both MemoryManager and DeviceManager are updated through this commit to
handle the creation of multiple virtio-mem devices if needed. For now,
only the framework is in place, but the behavior remains the same, which
means only the memory zone created from '--memory' generates a
virtio-mem region that can be used for resize.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to anticipate the need for storing memory regions along with
virtio-mem information for each memory zone, we create a new structure
MemoryZone that will replace Vec<Arc<GuestRegionMmap>> in the hash map
MemoryZones.
This makes thing more logical as MemoryZones becomes a list of
MemoryZone sorted by their identifier.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Inject CPUID leaves for advertising KVM HyperV support when the
"kvm_hyperv" toggle is enabled. Currently we only enable a selection of
features required to boot.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Currently we don't need to do anything to service these exits but when
the synthetic interrupt controller is active an exit will be triggered
to notify the VMM of details of the synthetic interrupt page.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The Windows virtio block driver puts multiple data descriptors between
the header and the status footer. To handle this when parsing iterate
over the descriptor chain until the end is reached accumulating the
address and length pairs in a vector. For execution iterate over the
vector and make sequential reads from the disk for each data descriptor.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In anticipation for supporting multiple virtio descriptors, let's make
sure the read/write operations are performed with vectored I/O.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Some of the io_uring setup happens upon activation of the virtio-blk
device, which is initially triggered through an MMIO VM exit. That's why
the vCPU threads must authorize io_uring related syscalls.
This commit ensures the virtio-blk io_uring implementation can be used
along with the seccomp filters enabled.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We observed CI instability for the past couple of days. This
instability is confirmed to be a result of incomplete seccomp
filters. Given the filter on 'virtio_vsock' is recently added and
is missing 'brk', it is likely to be the root cause of the
instability.
Signed-off-by: Bo Chen <chen.bo@intel.com>
Extract common code for adding devices to the PCI bus into its own
function from the VFIO and VIRTIO code paths.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This removes the dependency of the pci crate on the devices crate which
now only contains the device implementations themselves.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Extending the Cloud-Hypervisor CI to allow for testing SGX on a
dedicated machine where special image and kernels are ready.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The new function already checks if the API version is compatible. There
is no need to expose the get_api_version function to code outside
hypervisor crate.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
In our build-script (build.rs), we won't set the environment variable
'BUILD_VERSION' when the 'git describe' command failed (e.g. when the
current source tree does not contain git information). This patch added
a fall back path where the default value of 'BUILD_VERSION' is based on
the 'cloud-hypervisor' crate version.
Fixes: #1669
Signed-off-by: Bo Chen <chen.bo@intel.com>
The goal of this commit is to rename the existing NUMA option 'id' with
'guest_numa_id'. This is done without any modification to the way this
option behaves.
The reason for the rename is caused by the observation that all other
parameters with an option called 'id' expect a string to be provided.
Because in this particular case we expect a u32 representing a proximity
domain from the ACPI specification, it's better to name it with a more
explicit name.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the recent changes regarding how to describe NUMA nodes to the
guest, the documentation related to memory zones and NUMA has been
updated accordingly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Since both --memory-zone and --numa parameters have been updated with
addition and removal of multiple options, this commit updates the
related integration tests to ensure they are still valid.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The way to describe guest NUMA nodes has been updated through previous
commits, letting the user describe the full NUMA topology through the
--numa parameter (or NumaConfig).
That's why we can remove the deprecated and unused 'guest_numa_node'
option.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the previous changes introducing new options for both memory
zones and NUMA configuration, this patch changes the behavior of the
NUMA node definition. Instead of relying on the memory zones to define
the guest NUMA nodes, everything goes through the --numa parameter. This
allows for defining NUMA nodes without associating any particular memory
range to it. And in case one wants to associate one or multiple memory
ranges to it, the expectation is to describe a list of memory zone
through the --numa parameter.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This new option provides a new way to describe the memory associated
with a NUMA node. This is the first step before we can remove the
'guest_numa_node' option from the --memory-zone parameter.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that we have an identifier per memory zone, and in order to keep
track of the memory regions associated with the memory zones, we create
and store a map referencing list of memory regions per memory zone ID.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In anticipation for allowing memory zones to be removed, but also in
anticipation for refactoring NUMA parameter, we introduce a mandatory
'id' option to the --memory-zone parameter.
This forces the user to provide a unique identifier for each memory zone
so that we can refer to these.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Split the block device implementation into code that be used in common
between multiple different virtio device implementations.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In order to simplify the transition to VirtioCommon and to avoid needing
to set empty fields derive Default for VirtioCommon.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Rearrange the code to match other devices which makes it easier to prep
for sharing this between other devices.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Move the if-let for the taps later which makes the earlier activation
code identical to other devices.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Introduce VirtioCommon to help remove duplicated functionality and state
between implementations of VirtioDevice. Initially it is only handling
feature acknowledgement and testing.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This patch ported many tests to the new methodology, where the guest log
will be printed only when the test is failing.
Things to finish in follow-up PRs:
1. Special tests not ported yet include: test_reboot,
test_bzimage_reboot, test_serial_null(), test_serial_tty(),
test_serial_file(), test_virtio_console(), test_console_file(),
and test_simple_launch.
2. Few direct calls to 'Command::new(clh_command("cloud-hypervisor"))',
which is still printing the guest console
Signed-off-by: Bo Chen <chen.bo@intel.com>
Writing some new documentation to help users understand how the guest
memory can be described through Cloud-Hypervisor parameters.
Fixes#1659
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By extending the existing NUMA integration test, this commit validates
the proper distances between NUMA nodes are exposed to the guest.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By introducing the SLIT (System Locality Distance Information Table), we
provide the guest with the distance between each node. This lets the
user describe the NUMA topology with a lot of details so that slower
memory backing the VM can be exposed as being further away from other
nodes.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the NumaConfig which now provides distance information, we can
internally update the list of NUMA nodes with the exact distances they
should be located from other nodes.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By introducing 'distances' option, we let the user describe a list of
destination NUMA nodes with their associated distances compared to the
current node (defined through 'id').
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Extend the existing NUMA integration to validate that specifying CPUs
for each NUMA node gets propagated to the guest.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the list of CPUs related to each NUMA node, Processor Local
x2APIC Affinity structures are created and included into the SRAT table.
This describes which CPUs are part of each node.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Relying on the list of CPUs defined through the NumaConfig, this patch
will update the internal list of CPUs attached to each NUMA node.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Through this new parameter, we give users the opportunity to specify a
set of CPUs attached to a NUMA node that has been previously created
from the --memory-zone parameter.
This parameter will be extended in the future to describe the distance
between multiple nodes.
For instance, if a user wants to attach CPUs 0, 1, 2 and 6 to a NUMA
node, here are two different ways of doing so:
Either
./cloud-hypervisor ... --numa id=0,cpus=0-2:6
Or
./cloud-hypervisor ... --numa id=0,cpus=0:1:2:6
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This new test validates the guest OS can find the NUMA nodes which have
been defined by the user through the CLI, and that the right amount of
memory is associated with each node.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The SRAT table (System Resource Affinity Table) is needed to describe
NUMA nodes and how memory ranges and CPUs are attached to them.
For now it simply attaches a list of Memory Affinity structures based on
the list of NUMA nodes created from the VMM.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the 'guest_numa_node' option, we create and store a list of
NUMA nodes in the MemoryManager. The point being to associate a list of
memory regions to each node, so that we can later create the ACPI tables
with the proper memory range information.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
With the introduction of this new option, the user will be able to
describe if a particular memory zone should belong to a specific NUMA
node from a guest perspective.
For instance, using '--memory-zone size=1G,guest_numa_node=2' would let
the user describe that a memory zone of 1G in the guest should be
exposed as being associated with the NUMA node 2.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Given that ACPI uses u32 as the type for the Proximity Domain, we can
use u32 instead of u64 as the type for 'host_numa_node' option.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
"struct MemoryConfig" has balloon_size but not in MemoryConfig
of cloud-hypervisor.yaml.
This commit adds it.
Signed-off-by: Hui Zhu <teawater@antfin.com>
Let's narrow down the limitation related to mbind() by allowing shared
mappings backed by a file backed by RAM. This leaves the restriction on
only for mappings backed by a regular file.
With this patch, host NUMA node can be specified even if using
vhost-user devices.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Relying on the new option 'host_numa_node' from the 'memory-zone'
parameter, the user can now define which NUMA node from the host
should be used to back the current memory zone.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Since memory zones have been introduced, it is now possible for a user
to specify multiple backends for the guest RAM. By adding a new option
'host_numa_node' to the 'memory-zone' parameter, we allow the guest RAM
to be backed by memory that might come from a specific NUMA node on the
host.
The option expects a node identifier, specifying which NUMA node should
be used to allocate the memory associated with a specific memory zone.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The flag 'mergeable' should only apply to the entire guest RAM, which is
why it is removed from the MemoryZoneConfig as it is defined as a global
parameter at the MemoryConfig level.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The 'cmdline' parameter should not be required as it is not needed when
the 'kernel' parameter is the rust-hypervisor-fw, which means the kernel
and the associated command line will be found from the EFI partition.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Factorize the codepath between simple memory and multiple memory zones.
This simplifies the way regions are memory mapped, as everything relies
on the same codepath. This is performed by creating a memory zone on the
fly for the specific use case where --memory is used with size being
different from 0. Internally, the code can rely on memory zones to
create the memory regions forming the guest memory.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
After the introduction of user defined memory zones, we can now remove
the deprecated 'file' option from --memory parameter. This makes this
parameter simpler, letting more advanced users define their own custom
memory zones through the dedicated parameter.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
User defined memory regions can now support being snapshot and restored,
therefore this commit removes the restrictions that were applied through
earlier commit.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By factorizing a lot of code into create_ram_region(), this commit
achieves the simplification of the restore codepath. Additionally, it
makes user defined memory zones compatible with snapshot/restore.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
First thing, this patch introduces a new function to identify if a file
descriptor is linked to any hard link on the system. This can let the
VMM know if the file can be accessed by the user, or if the file will
be destroyed as soon as the VMM releases the file descriptor.
Based on this information, and associated with the knowledge about the
region being MAP_SHARED or not, the VMM can now decide to skip the copy
of the memory region content. If the user has access to the file from
the filesystem, and if the file has been mapped as MAP_SHARED, we can
consider the guest memory region content to be present in this file at
any point in time. That's why in this specific case, there's no need for
performing the copy of the memory region content into a dedicated file.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Let's not assume that a backing file is going to be the result from
a snapshot for each memory region. These regions might be backed by
a file on the host filesystem (not a temporary file in host RAM), which
means they don't need to be copied and stored into dedicated files.
That's why this commit prepares for further changes by introducing an
optional PathBuf associated with the snapshot of each memory region.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
There will be some cases where the implementation of the snapshot()
function from the Snapshottable trait will require to modify some
internal data, therefore we make this possible by updating the trait
definition with snapshot(&mut self).
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Adding a small test to validate that user defined memory zones work as
expected when using --memory-zone option.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In case the memory size is 0, this means the user defined memory
zones are used as a way to specify how to back the guest memory.
This is the first step in supporting complex use cases where the user
can define exactly which type of memory from the host should back the
memory from the guest.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In anticipation for the need to map part of a file with the function
create_ram_region(), it is extended to accept a file offset as argument.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In case the provided backing file is an actual file and not a directory,
we should not truncate it, as we expect the file to already be the right
size.
This change will be important once we try to map the same file through
multiple memory mappings. We can't let the file be truncated as the
second mapping wouldn't work properly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Introducing a new CLI option --memory-zone letting the user specify
custom memory zones. When this option is present, the --memory size
must be explicitly set to 0.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
It is otherwise seems to be able to cause resource conflicts with
Windows APCI_HAL. The OS might do a better job on assigning resources
to this device, withouth them to be requested explicitly. 0xcf8 and
0xcfc are only what is certainly needed for the PCI device enumeration.
Signed-off-by: Anatol Belski <anatol.belski@microsoft.com>
Use of backing file is deprecated hence use the `hugepages` field.
Also use the `boot` field for specifying number of cpus
Signed-off-by: Amey Narkhede <ameynarkhede02@gmail.com>
We may need to store hypervisor speciific data to the VM. This support is
needed for Microsoft hyperv implementations. This patch introduces two
new definitions to Vm trait and implements for KVM.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Problem with the previous solution was that Cleanup stage was not
executed when previous stages failed. We fix this by adding a post
section that executes always after the Aarch64 build completed, no
matter if it has failed, succeeded or been aborted.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
"debug!" marco is used in virtio-devices/src/epoll_helper.rs. When"-vvv"
and "--log-file" option was specified, the missing "SYS_write" rule
caused a "bad system call" crash.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Cleanup of the Aarch64 machine can't be done as part of the parallel
stage as this is often skipped. When the build is aborted because
another parallel stage failed, the post actions are simply not
performed. That's why we need a dedicated stage, out of the parallel
ones, to cleanup the Aarch64 machine.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Use updated --memory option parameters. The field `size` needs
M/G suffix. Without the suffix cloud-hypervisor panics at
src/main.rs:353
Also the use of backing file is deprecated so use `shared` field
Signed-off-by: Amey Narkhede <ameynarkhede02@gmail.com>
As we never join the spawned virtio-devices worker threads, the error
returned from each worker thread is lost. For now, we simply print out
the error from each worker thread.
Fixes: #1551
Signed-off-by: Bo Chen <chen.bo@intel.com>
From the experiments of running integration tests on my local machine,
auditd occationally reported the 'brk' syscall is needed for the
'virtio-rng' worker thread.
Signed-off-by: Bo Chen <chen.bo@intel.com>
This patch adds the seccomp filter list for the virtio_net thread, while
the list was already added for the virtio_net_ctl thread.
Partially fixes: #925
Signed-off-by: Bo Chen <chen.bo@intel.com>
The current seccomp filter for virtio-net is actually for the worker
thread 'virtio_net_ctl' (not the actual worker thread
'virtio_net'). This patch introduces changes to distinguish those two
worker threads and seccomp filters.
Signed-off-by: Bo Chen <chen.bo@intel.com>
We recently added the ability for the gnu and musl workers to retry if
integration test were not passing, relying on some simple Jenkins
options. Unfortunately, this is not working as expected as the retries
never pass either. The suspected reason is the machine itself, which
might be scheduled on some specific hardware, which makes our VMs more
error prone.
Bottom line, on a faulty machine, the tests will always fail, therefore
there is no added value in retrying on the same machine.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Some OS might check for duplicates and bail out, if it can't create a
distinct mapping. According to ACPI 5.0 section 6.1.12, while _UID is
optional, it becomes required when there are multiple devices with the
same _HID.
Signed-off-by: Anatol Belski <ab@php.net>
Using the Rust Barrier mechanism, this patch forces each virtio device
to acknowledge they've been correctly paused before going further.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
If the build was aborted then the dev_cli.sh code that is responsible
for changing the file ownership will not get run. This results in the
failure to delete some of the files in the workspace.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The seccomp filters specific to the virtio-net threads must contain
dup() syscall now that we ported the epoll code to the EpollHelper.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Migrate virtio-net and vhost-user-net control queue to EpollHelper so
as to remove code duplication.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This required a bit of rearranging as it is not possible to call
prepare_daemon() inside a catch_unwind{} block.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Migrate all vhost-user devices to EpollHelper so as to remove code that
is duplicated between multiple virtio devices.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now the testing aspects are run inside a panic handler block rather than
inside a credibility TestBlock. If the test fails then the output from
the cloud-hypervisor binary is then presented.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This reduces the complexity of the test slightly. The PCI BDFs in the L1
needed changing as the block devices come before the network ones.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Migrate to EpollHelper so as to remove code that is duplicated between
multiple virtio devices.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Instead of passing only the event type through the handle_event()
callback, we make the trait slightly more generic by providing the
epoll event to each virtio device implementation.
This is particularly useful for vsock as it will need the event set.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Migrate to EpollHelper so as to remove code that is duplicated between
multiple virtio devices.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The seccomp filters specific to the virtio-console thread must contain
dup syscall now that we ported the epoll code to the EpollHelper.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Migrate to EpollHelper so as to remove code that is duplicated between
multiple virtio devices.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The brk syscall is not always called as the system might not need it.
But when it's needed from the API thread, this causes the thread to
terminate as it is not part of the authorized list of syscalls.
This should fix some sporadic failures on the CI with the musl build.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Even if the vhost-user-net device did implement all currently-defined
features, it would be very short-sighted to use ::all(), because if a
new feature was defined later, the device would start claiming to
implement it even though it didn't.
More practically, claiming to implement all features breaks using QEMU
with the cloud-hypervisor vhost-user-net backend, because QEMU will
negotiate VHOST_USER_PROTOCOL_F_SLAVE_REQ, and then break when the
communication channel isn't actually set up.
I wasn't sure exactly which features the backend should claim to
implement, though. Definitely MQ, and I'm fairly certain none of the
features I've ommitted are implemented. But I'm not sure about
REPLY_ACK. As far as I can tell it should be implemented entirely by
the vhost crate, with no cooperation required from the vhost-user-net
backend itself, so there should be no reason to let a frontend use it
if it wants to. But despite this, neither vhost-user-fs nor
vhost-user-blk claims to implement it.
Signed-off-by: Alyssa Ross <hi@alyssa.is>
Rather than using a credible TestBlock to capture the test assertions
instead use a catch_unwind block to catch the panic and turn
it into a Result<>.
If block panicked or the child binary had non-zero exit then, and only
then, print out the child output.
This results in a clearer test output with no interleaving.
Currently only test_counters is ported to this methodology to
demonstrate its use.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
`cargo test` runs every test in its own process and by default will
collect the output from the test and only print it if the test fails.
`--nocapture` instead prints any output from the test immediately and
does not collate it - this leads to interleaving.
`--nocapture` might be useful for local test debugging but is not
something that is helpful, and is instead unhelpful for our CI as a
whole.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Now that io-uring crate has been stabilized, let's move to the latest
release 0.4.0 from crates.io.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Migrate to EpollHelper so as to remove code that is duplicated between
multiple virtio devices.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Packaging the cloud-hyperviosr binary is useful for enterprise use.
This patch adds the spec file for x86_64. Aarch64 support is not added
yet.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Both gnu and musl workers will retry integration tests up to 3 times if
they fail. This should give us a better pass rate, without having to
restart the entire build every time a single glitch happens.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Add mprotect to API thread rules. Prevent the VMM is
killed when it is used.
Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
This fixes `cargo vendor` throwing an error
```
$ cargo vendor
error: failed to sync
Caused by:
found duplicate version of package `vfio-bindings v0.2.0` vendored from two sources:
source 1: https://github.com/rust-vmm/vfio-bindings#f08cbcbf
source 2: registry `https://github.com/rust-lang/crates.io-index`
```
Both sources are indeed same, the conflict is only cause by the
different URLs.
Signed-off-by: Anatol Belski <ab@php.net>
Based on the way KVM_GET_MSRS and KVM_SET_MSRS work, both function are
very unlikely to fail, as they simply stop looping through the list of
MSRs as soon as getting or setting one fails. This is causing some
issues with the snapshot/restore feature, as on some platforms, we only
save a subset of the list of MSRs, leading to unproper way of saving the
VM.
The way to address this issue is by checking the number of MSRs get/set
matches the expected amount from the list. In case it does not match, we
simply ignore the failing MSR and continue getting/setting the rest of
the list. By doing this by iterations, we end up getting/setting as many
MSRs as the platform can support.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Turns out that "when" blocks are permitted but not effectual in "post"
sections so instead use a script behaviour to make the message
conditional on build change.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Migrate to EpollHelper so as to remove code that is duplicated between
multiple virtio devices.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Currently any messages generated during the worker thread are not
shown anywhere as the thread is never join()ed on. Instead output the
error immediately.
For now only cover the subset where the work to port to EpollHandler
clashed with the seccomp filtering for virtio devices.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
We no longer download focal-server-cloudimg-arm64.img, checking it
lead to a failure if "work_loads" folder is new.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Migrate to EpollHelper so as to remove code that is duplicated between
multiple virtio devices.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Migrate to EpollHelper so as to remove code that is duplicated between
multiple virtio devices.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Increase the number of open files limit for the sandboxed process to the
maximum allowed in the system. The maximum is obtained by reading the
/proc/sys/fs/nr_open sysctl file, and the setting is done using the setrlimit
syscall. Failure to read or parse the nr_open file, or to set the rlimit
results in a panic.
Signed-off-by: Ricardo Koller <ricarkol@gmail.com>
This commit enables the test case for testing the basic function
of virtio_vsock (i.e. without the hotplug).
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This patch added the seccomp_filter module to the virtio-devices crate
by taking reference code from the vmm crate. This patch also adds
allowed-list for the virtio-block worker thread.
Partially fixes: #925
Signed-off-by: Bo Chen <chen.bo@intel.com>
This patch propagates the SeccompAction value from main to the
Vm struct constructor (i.e. Vm::new_from_memory_manager), so that we can
use it to construct the DeviceManager and CpuManager struct for
controlling the behavior of the seccomp filters for vcpu/virtio-device
worker threads.
Signed-off-by: Bo Chen <chen.bo@intel.com>
This patch extends the CLI option '--seccomp' to accept the 'log'
parameter in addition 'true/false'. It also refactors the
vmm::seccomp_filters module to support both "SeccompAction::Trap" and
"SeccompAction::Log".
Fixes: #1180
Signed-off-by: Bo Chen <chen.bo@intel.com>
This patch replaces the usage of 'SeccompLevel' with 'SeccompAction',
which is the first step to support the 'log' action over system
calls that are not on the allowed list of seccomp filters.
Signed-off-by: Bo Chen <chen.bo@intel.com>
This commit enables the `api_create_boot` case in the integration
test as the test for the Cloud Hypervisor API server functionality.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
By adding a new io_uring feature gate, we let the user the possibility
to choose if he wants to enable the io_uring improvements or not.
Since the io_uring feature depends on the availability on recent host
kernels, it's better if we leave it off for now.
As soon as our CI will have support for a kernel 5.6 with all the
features needed from io_uring, we'll enable this feature gate
permanently.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Instead of just logging error messages but continue the processing of
the queues, this patch returns errors right away. This allows for a
quicker detection of an error happening on the virtqueue.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In case the host supports io_uring and the specific io_uring options
needed, the VMM will choose the asynchronous version of virtio-blk.
This will enable better I/O performances compared to the default
synchronous version.
This is also important to note the VMM won't be able to use the
asynchronous version if the backend image is in QCOW format.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This introduces a new version of virtio-blk device. The default
virtio-blk provides synchronous processing of the queues, while this
new version relies on io_uring from the host kernel to provide an
asynchronous processing of the queues.
This new asynchronous version provides a huge performance improvement
compared to the default synchronous version.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Creates a dedicated function relying on io_uring crate to execute
io_uring specific requests.
Also creates a function for checking io_uring support on the host.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
PR#1511 introduced a `flock` command in order to let AArch64 CI
can be executed with multiple executors. However the command
```
(
echo "try to lock $WORKLOADS_DIR folder and update"
flock -x 12 && update_workloads
)
```
will introduce an abnormal behavior: If any error happened in
function `updated_workloads`, the sub-shell opened by the pair of
parentheses will be killed instead of the main shell, which is not
right.
This commit fixes this abnormal execution behaviour.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Cloud Hypervisor allows either the serial or virtio console to output to
TTY, but TTY input is pushed to both.
This is not correct. When Linux guest is configured to spawn TTYs on
both ttyS0 and hvc0, the user effectively issues the same commands twice
in different TTYs.
Fix this by only direct input to the one choice that is using host side
TTY.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Lock "work_loads" folder when one job is syncing files. If another job
arrives, wait until the lock is released.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
The command which is mentioned in the README is wrong. We have two
cofigs one for x86_64 and another for aarh64. Previously it was a single
config. After adding the configs the read me was not modified. This
patch fixes the issue.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
In order to follow the virtio-fs development, we rely on the latest
development branch regarding the virtio-fs daemon. It will be now build
from virtio-fs-dev branch from the virtio-fs maintainers repository.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Move to latest vhost version, as it contains the updated list of
vhost-user protocol features along with the updated list of virtio-fs
slave commands.
This will allow Cloud-Hypervisor to work with latest virtiofsd binary.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit fixes an "Bad syscall" error when shutting down the VM
on AArch64 by adding the SYS_unlinkat syscall to the seccomp
whitelist.
Signed-off-by: Jianyong Wu <jianyong.wu@arm.com>
EpollHelper allows the removal of much duplicated loop handling code and
instead the device specific even handling is delegated via an
implementation of EpollHelperHandler.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This is required for implementing virtio-net as the epoll RawFd must be
assigned into the NetQueuePair.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Ensure that the width of the I/O port is correctly set to 32-bits in the
generic address used for the X_PM_TMR_BLK. Do this by type
parameterising GenericAddress::io_port_address() fuction.
TEST=Boot with clocksource=acpi_pm and observe no errors in the dmesg.
Fixes: #1496
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Port virtio-block device to the new EpollHelper. This required moving
the queue EventFd ownership to BlockEpollHandler.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This is a helper for implementing the worker thread for virtio devices
and in particular handles special behaviour for pause and kill events.
The device specific event handling (for the queues themselves) is
delegated to a method invoked on a new EpollHelperHandler trait. This
method is passed the event as well as the EpollHelper so that it can
operate on the handler in order to manage events itself (required for
virtio-net.)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This is a counter exposed via an I/O port that runs at 3.579545MHz. Here
we use a hardcoded I/O and expose the details through the FADT table.
TEST=Boot Linux kernel and see the following in dmesg:
[ 0.506198] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add some documentation regarding the snapshot/restore feature which was
introduced earlier, explaining how the feature works and how to use it
properly.
Fixes#1490
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This bit should be set for the last 224us of each second. To be able to
fill this bit we need to use clock_gettime() to get the nanoseconds.
Also set the frequency bit to indicate we update at 32kHz.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
We store the device passthrough handler, so we should use it through our
internal API and only carry the passed through device configuration.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Shrink GICDevice trait to contain hypervisor agnostic API's only, which
are used in generating FDT.
Move all KVM specific logic into KvmGICDevice trait.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
This commit enables some mmio-related integration test cases on
AArch64, including:
* some vhost_user test cases
* virtio-blk test cases
* pmem test cases
Also this commit contains a bug fix in creating virtio-blk device.
Previously, when creating the FDT, the virtio-blk device was
labeled in the reverse order of address allocation.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Make set_gsi_routing take a list of IrqRoutingEntry. The construction of
hypervisor specific structure is left to set_gsi_routing.
Now set_gsi_routes, which is part of the interrupt module, is only
responsible for constructing a list of routing entries.
This further splits hypervisor specific code from hypervisor agnostic
code.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
That function is going to return a handle for passthrough related
operations.
Move create_kvm_device code there.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Add the basic infrastructure for fuzzing along with a qcow fuzzer ported
from crosvm and adapted to our code.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The only driver writable field in the virtio-block specification is the
writeback one. Check that the offset being written to is for that field
and update it.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The only driver writable field in the virtio-block specification is the
writeback one. Check that the offset being written to is for that field
and update it.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add a helper function to share code between implementations that can use
a slice accessible data structure for configuration data.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Remove the write_config() implementations that only generate a warning
as that is now done at the VirtioDevice level.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Not every virtio device has any config fields that can be read and most
have none that can be written to.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The retry order to create virtual GIC is GICv3-ITS, GICv3 and GICv2.
But there was not log message to show what was finally created.
The log messages also mute the warning for unused "log" crate.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
With recent changes (#1030#1084), the '/vm.remove-device' API is no
longer restricted to remove VFIO PCI devices. It now supports (almost)
all hotpluggable devices, e.g. disk/net/pmem/fs/vsock.
Signed-off-by: Bo Chen <chen.bo@intel.com>
It returns an hypervisor object depending on which hypervisor is
configured. Currently it only supports KVM.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
SGX expects the EPC region to be reported as "reserved" from the e820
table. This patch adds a new entry to the table if SGX is enabled.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The support for SGX is exposed to the guest through CPUID 0x12. KVM
passes static subleaves 0 and 1 from the host to the guest, without
needing any modification from the VMM itself.
But SGX also relies on dynamic subleaves 2 through N, used for
describing each EPC section. This is not handled by KVM, which means
the VMM is in charge of setting each subleaf starting from index 2
up to index N, depending on the number of EPC sections.
These subleaves 2 through N are not listed as part of the supported
CPUID entries from KVM. But it's important to set them as long as index
0 and 1 are present and indicate that SGX is supported.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Instead of passing the GuestMemoryMmap directly to the CpuManager upon
its creation, it's better to pass a reference to the MemoryManager. This
way we will be able to know if SGX EPC region along with one or multiple
sections are present.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The SGX EPC region must be exposed through the ACPI tables so that the
guest can detect its presence. The guest only get the full range from
ACPI, as the specific EPC sections are directly described through the
CPUID of each vCPU.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the presence of one or multiple SGX EPC sections from the VM
configuration, the MemoryManager will allocate a contiguous block of
guest address space to hold the entire EPC region. Within this EPC
region, each EPC section is memory mapped.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Introducing the new CLI option --sgx-epc along with the OpenAPI
structure SgxEpcConfig, so that a user can now enable one or multiple
SGX Enclave Page Cache sections within a contiguous region from the
guest address space.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Extract the code that is used by vhost_user_block from the
virtio-devices crate to remove the dependencies on unrequired
functionality such as the virtio transports.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This commit adds required environment configurations to the
`dev_cli.sh` and a Jenkins stage to enable AArch64 binary
building using musl toolchain.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
As the current AArch64 integration test builds kernel every time,
which would take unnecessary time in CI and therefore not ideal.
This commit refactors the AArch64 kernel building strategy to:
1. Keep the Linux kernel source code directory instead of deleting
it everytime after the kernel is built.
2. In integration test script, check if the Linux kernel source
code directory exists. If so use `git fetch && git checkout -f` to
keep the source code always updated, else run `git clone` to get
the source code.
3. Copy config file in and then compile the kernel.
Fixes: https://github.com/cloud-hypervisor/cloud-hypervisor/issues/1444
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This simplies some of the handling for PCI BARs particularly with
respect to snapshot and restore. No attempt has been made to handle the
64-bit bar handling in a different manner to that which was used before.
Fixes: #1153
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In this commit we saved the BDF of a PCI device and set it to "devid"
in GSI routing entry, because this field is mandatory for GICv3-ITS.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
The type is now hypervisor::Vm. Switch from KVM specific name vm_fd to a
generic name just like 8186a8eee6 ("vmm: interrupt: Rename vm_fd").
No functional change.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Under high load, the VM might take some time to hotplug the disk after
the hotplug command has been issued. For this reason, let's put a 10s
sleep before checking for the presence of the new disk.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We want to give the time to the nested VM to be fully ready before we
check it's correctly setup. This involves 3 layers of virtualization
when running the CI on Azure, which added to the high load happening
because of the parallelization, adds up to the start up time.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The virtio configuration structures have been slightly modified between
5.6-rc4 and 5.8-rc4, forcing the virtio-iommu device to be updated
accordingly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Updating the kernel from 5.6-rc4 to 5.8-rc4 allows us to remove the
dependency on both virtio-vsock and virtio-mem patches as they are now
part of the upstream kernel. We're still carrying virtio-iommu and
virtio-fs patches.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Move the definition of RawFile from virtio-devices crate into qcow
crate. All the code that consumes RawFile also already depends on the
qcow crate for image file type detection so this change breaks the
need for the qcow crate to depend on the very large virtio-devices
crate.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
As compiling without acpi (implied by mmio) means that the VM will
terminate on i8042 reset we cannot test the reboot.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Now that vhost_user_net crate does not depend on the virtio-devices
crate it is no longer compiled differently based on the mmio or pci
features.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Move NetQueuePair and the related NetCounters into the net_util crate.
This means that the vhost_user_net crate now no longer depends on
virtio-devices and so does not depend on the pci, qcow or other similar
crates. This significantly simplifies the build chain for this backend.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
By moving the code for opening the two RX and TX queues into a shared
location we are starting to remove the requirement for the
vhost-user-net backend to depend on the virtio-devices crate which in of
itself depends on many other crates that are not necessary for the
backend to function.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Rather than use an embedded String inside the MultiQueueSupport error
value use two different values to differentiate the two cases.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
By moving the code for opening the TAP device into a shared location we
are starting to remove the requirement for the vhost-user-net backend to
depend on the virtio-devices crate which in of itself depends on many
other crates that are not necessary for the backend to function.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
It gets bubbled all the way up from hypervsior crate to top-level
Cargo.toml.
Cloud Hypervisor can't function without KVM at this point, so make it
a default feature.
Fix all scripts that use --no-default-features.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Check if increasing the time after the VM is spawned help with getting
more stable numbers for the base PSS.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit store balloon size to MemoryConfig.
After reboot, virtio-balloon can use this size to inflate back to
the size before reboot.
Signed-off-by: Hui Zhu <teawater@antfin.com>
The implementation of the virtio-balloon was slightly wrong as it was
generating the GPA (Guest Physical Address) from the PFN (Page Frame
Number) which was a u32. That means the GPA was created as a u32, and
later a cast was done to extend it to a u64 type. Unfortunately, by
doing so, the GPA was wrong if the value was supposedly more than 32
bits.
That's why the PFN is casted into a u64 before the GPA is generated,
which creates the GPA on 64 bits directly.
Additionally, this patch simplifies the process_queue() function,
relying on multiple vm-memory helpers.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit adds a Jenkins stage for AArch64 integration test,
and the test is carried out using the GNU toolchain.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
The compiled AArch64 linux kernel by running `make` is in PE format
instead of vmlinux, vmlinux.pvh and bzImage format. Therefore we
need to add integration test for this case.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This commit adds supporting components and code for enabling the
AArch64 integration tests, including:
1. A Linux kernel config file to build kernel on AArch64 machines.
2. Refactoring the `run_integration_test.sh` to architecture
specific scripts for readability.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Remove the vmm dependency from vhost_user_block and vhost_user_net where
it was existing to use config::OptionParser. By moving the OptionParser
to its own crate at the top-level we can remove the very heavy
dependency that these vhost-user backends had.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Refactored the construction of KVM IOCTL rules for Seccomp.
Separating the rules by architecture can reduce the risk of bugs and
attacks.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
In order to move the hypervisor specific parts of the VM exit handling
path, we're defining a generic, hypervisor agnostic VM exit enum.
This is what the hypervisor's Vcpu run() call should return when the VM
exit can not be completely handled through the hypervisor specific bits.
For KVM based hypervisors, this means directly forwarding the IO related
exits back to the VMM itself. For other hypervisors that e.g. rely on the
VMM to decode and emulate instructions, this means the decoding itself
would happen in the hypervisor crate exclusively, and the rest of the VM
exit handling would be handled through the VMM device model implementation.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Fix test_vm unit test by using the new abstraction and dropping some
dead code.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
The code is purely for maintaining an internal counter. It is not really
tied to KVM.
No functional change.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
There are several dependencies that need updating so update them
manually rather than relying on dependabot. This will reduce the load on
the CI.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The binary is still built in the same location but the source code and
the dependencies for it come from the vhost_user_block crate itself.
The binary will be built with:
`cargo build --all --bin vhost_user_block` or just `cargo build --all`
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The binary is still built in the same location but the source code and
the dependencies for it come from the vhost_user_net crate itself.
The binary will be built with:
`cargo build --all --bin vhost_user_net` or just `cargo build --all`
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
With vhost_user_fs binary moved to its own crate the dependencies in the
top level can be trimmed significantly.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The binary is still built in the same location but the source code and
the dependencies for it come from the vhost_user_fs crate itself.
The binary will be built with:
`cargo build --all --bin vhost_user_fs` or just `cargo build --all`
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In preparation for splitting the binaries into their own crates start
building all the binaries in the workspace as part of the integration
testing suite.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In preparation for splitting the binaries into their own crates start
building all the binaries in the workspace when running the build
command inside the container.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In preparation for splitting the binaries into their own crates start
building all the binaries in the workspace when doing a build as part of
the GitHub actions.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The _fd suffix is KVM specific. But since it now point to an hypervisor
agnostic hypervisor::Vm implementation, we should just rename it vm.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The _fd suffix is KVM specific. But since it now point to an hypervisor
agnostic hypervisor::Vm implementation, we should just rename it vm.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The fd naming is quite KVM specific. Since we're now using the
hypervisor crate abstractions, we can rename those into something more
readable and meaningful.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The fd naming is quite KVM specific. Since we're now using the
hypervisor crate abstractions, we can rename those into something more
readable and meaningful.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The fd naming is quite KVM specific. Since we're now using the
hypervisor crate abstractions, we can rename those into something more
readable and meaningful. Like e.g. vcpu or vm.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Now that our CI has transitioned from ClearLinux to Ubuntu images
exclusively, let's update the documentation to refer to Ubuntu images
instead of ClearLinux's ones.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that the project relies on Ubuntu images, the documentation related
to the creation of a custom image has been updated. It's important to
note this procedure could be applied to any other distribution, given
that the package manager's commands would be slightly different.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to differentiate tests that can be run in parallel versus
tests that must be run on their own, we move all tests into dedicated
modules.
The point is to avoid glitches in results that can be caused by the fact
that other tests (hence VMs) are running at the same time.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Split the generic virtio code (queues and device type) from the
VirtioDevice trait, transport and device implementations.
This also simplifies the feature handling in vhost_user_backend as the
vm-virtio crate is no longer has any features.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Move the CI to rely entirely on Ubuntu cloud images. It's worth noting
that both QCOW2 and RAW images from Ubuntu Focal Fossa have been
modified to include the tools needed from integration tests.
This means fio, iperf, iperf3, netcat and socat have been added to the
image. The snapd package have been fully removed as it was expecting the
support for squashfs (not present when using our own kernel from direct
kernel boot), which was causing some failures, and was preventing
cloud-init from terminating properly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Updating the cloud-init files related to Ubuntu images is needed to be
able to use Ubuntu images for running all integration tests.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
With QCOW disk images the space needed is greater than the size of the
iamge as any "zero" blocks in the image are allocated when they are
touched making the image bigger.
Here we add a threshold of 6GiB with added debugging messages to ensure
that there is sufficient disk space to run the tests.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This update of Dockerfile will add support to the AArch64
integration tests and musl building, including:
1. Installation of the missing `setcap` binary on AArch64.
On AArch64, `setcap` binary should be installed via `libcap2-bin`.
This binary is needed in the integration test.
2. Related support for the cloud-hypervisor binary building by
musl toolchain.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Extend the existing integration test test_snapshot_restore by testing
with more than one vCPU.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
On x86 architecture, we need to save a list of MSRs as part of the vCPU
state. By providing the full list of MSRs supported by KVM, this patch
fixes the remaining snapshot/restore issues, as the vCPU is restored
with all its previous states.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Add a new function to the hypervisor trait so that the caller can
retrieve the list of MSRs supported by this hypervisor.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Some vCPU states such as MP_STATE can be modified while retrieving
other states. For this reason, it's important to follow a specific
order that will ensure a state won't be modified after it has been
saved. Comments about ordering requirements have been copied over
from Firecracker commit 57f4c7ca14a31c5536f188cacb669d2cad32b9ca.
This patch also set the previously saved VCPU_EVENTS, as this was
missing from the restore codepath.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The logic can be shared among hypervisor implementations.
The 'static bound is used such that we don't need to deal with extra
lifetime parameter everywhere. It should be okay because we know the
entry type E doesn't contain any reference.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
This trait contains a function which produces a interrupt routing entry.
Implement that trait for KvmRoutingEntry and rewrite the update
function.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
The observation is only the route entry is hypervisor dependent.
Keep a definition of KvmMsiInterruptManager to avoid too much code
churn.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
The observation is that only the route field is hypervisor specific.
Provide a new function in blanket implementation. Also redefine
KvmRoutingEntry with RoutingEntry to avoid code churn.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
The observation is that the GSI hashmap remains untouched before getting
passed into the MSI interrupt manager. We can create that hashmap
directly in the interrupt manager's new function.
The drops one import from the interrupt module.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
The OVMF firmware loops around looking for an entry marking the end of
the table. Without this entry processing the tables is an infinite loop.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Taken from crosvm: 44336b913126d73f9f8d6854f57aac92b5db809e and adapted
for Cloud Hypervisor.
This is basic and incomplete support but Linux correctly finds the DMI
data based on this:
root@clr-c6ed47bc1c9d473d9a3a8bddc50ee4cb ~ # dmesg | grep -i dmi
[ 0.000000] DMI: Cloud Hypervisor cloud-hypervisor, BIOS 0
root@clr-c6ed47bc1c9d473d9a3a8bddc50ee4cb ~ # dmesg | grep -i smbio
[ 0.000000] SMBIOS 3.2.0 present.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Update the API documentation to reflect that the hotplug APIs return
data about the device as well as the newly added /vm.counters API.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This is a hash table of string to hash tables of u64s. In JSON these
hash tables are object types.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Initially the licensing was just Apache-2.0. This patch changes
the licensing to dual license Apache-2.0 OR BSD-3-Clause
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Each virtio thread was reading/draining the pause_evt pipe when
detecting the associated event. Problem is, when a virtio device has
multiple threads, they all share the same pause_evt pipe, which can
prevent some threads from receiving the event. If the first thread to
catch the event is quickly clearing the pipe, some other threads might
simply miss the event and they will not enter the "paused" state as
expected.
This is a behavior that was spotted with virtio-net as it usually uses
2 threads by default (1 for TX/RX queues and 1 for the control queue).
The way to solve this issue is by letting each thread drain the pipe
during the resume codepath, that is after the thread has been unparked.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Because we don't want the guest to miss any event triggered by the
emulation of devices, it is important to resume all vCPUs before we can
resume the DeviceManager with all its associated devices.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We need consistency between pause/resume and snapshot/restore
operations. The symmetrical behavior of pausing/snapshotting
and restoring/resuming has been introduced recently, and we must
now ensure that no matter if we're using pause/resume or
snapshot/restore features, the resulting VM should be running in
the exact same way.
That's why the vCPU state is now stored upon VM pausing. The snapshot
operation being a simple serialization of the previously saved state.
The same way, the vCPU state is now restored upon VM resuming. The
restore operation being a simple deserialization of the previously
restored state.
It's interesting to note that this patch ensures time consistency from a
guest perspective, no matter which clocksource is being used. From a
previous patch, the KVM clock was saved/restored upon VM pause/resume.
We now have the same behavior for TSC, as the TSC from the vCPUs are
saved/restored upon VM pause/resume too.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Instead of calling the resume() function from the CpuManager, which
involves more than what is needed from the shutdown codepath, and
potentially ends up with a deadlock, we replace it with a subset.
The full resume operation is reserved for a VM that has been paused.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We want each Vcpu to store the vCPU state upon VM pausing. This is the
reason why we need to explicitly implement the Pausable trait for the
Vcpu structure.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
When set_user_memory_region was moved to hypervisor crate, it was turned
into a safe function that wrapped around an unsafe call. All but one
call site had the safety statements removed. But safety statement was
not moved inside the wrapper function.
Add the safety statement back to help reasoning in the future. Also
remove that one last instance where the safety statement is not needed .
No functional change.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
That removes one more KVM-ism in VMM crate.
Note that there are more KVM specific code in those files to be split
out, but we're not at that stage yet.
No functional change.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Add a simple test to check that the data from the counters matches what
is expected and that the value of the counters increases after an
operation that will hit all counters.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Collate the virtio device counters in DeviceManager for each device that
exposes any and expose it through the recently added HTTP API.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The counters are a hash of device name to hash of counter name to u64
value. Currently the API is only implemented with a stub that returns an
empty set of counters.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Through the counters() function on the trait expose the accumulated
counters.
TEST=Observe that the counters from the VM match those from the tap on
the host (RX-TX inverted) and inside the guest (non inverted.)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The counters are a hash of counter name to (wrapping) u64 value. The
interpretation layer is responsible for converting this data into a
rate.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This corresponds to QEMU's 63659fe74e76f5c52854 commit.
the setattr code uses both fchmod and fchmodat in different cases,
however we only had fchmodat in the whitelist.
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
The structure is tightly coupled with KVM. It uses KVM specific
structures and calls. Add Kvm prefix to it.
Microsoft hypervisor will implement its own interrupt group(s) later.
No functional change intended.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Now that the VMM uses KVM_KVMCLOCK_CTRL from the KVM API, it must be
added to the seccomp filters list.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Through the newly added API notify_guest_clock_paused(), this patch
improves the vCPU pause operation by letting the guest know that each
vCPU is being paused. This is important to avoid soft lockups detection
from the guest that could happen because the VM has been paused for more
than 20 seconds.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
It's important that on restore path, the CpuManager's vCPU gets filled
with each new vCPU that is being created. In order to cover both boot
and restore paths, the list is being filled from the common function
create_vcpu().
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that the VMM uses both KVM_GET_CLOCK and KVM_SET_CLOCK from the KVM
API, they must be added to the seccomp filters list.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to maintain correct time when doing pause/resume and
snapshot/restore operations, this patch stores the clock value
on pause, and restore it on resume. Because snapshot/restore
expects a VM to be paused before the snapshot and paused after
the restore, this covers the migration use case too.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
When a request is made to increase the number of vCPUs in the VM attempt
to reuse any previously removed (and hence inactive) vCPUs before
creating new ones.
This ensures that the APIC ID is not reused for a different KVM vCPU
(which is not allowed) and that the APIC IDs are also sequential.
The two key changes to support this are:
* Clearing the "kill" bit on the old vCPU state so that it does not
immediately exit upon thread recreation.
* Using the length of the vcpus vector (the number of allocated vcpus)
rather than the number of active vCPUs (.present_vcpus()) to determine
how many should be created.
This change also introduced some new info!() debugging on the vCPU
creation/removal path to aid further development in the future.
TEST=Expanded test_cpu_hotplug test.
Fixes: #1338
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
After the vCPU has been ejected and the thread shutdown it is useful to
clear the "kill" flag so that if the vCPU is reused it does not
immediately exit upon thread recreation.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
These messages are intended to be useful to support debugging related to
vCPU hotplug/unplug issues.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Now that snapshot/restore is symmetrical, that is the VM must be paused
before it is snapshot and it must be resumed after it's been restored,
the integration test is updated accordingly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The same way the VM and the vCPUs are restored in a paused state, all
devices associated with the device manager must be restored in the same
paused state.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Because we need to pause the VM before it is snapshot, it should be
restored in a paused state to keep the sequence symmetrical. That's the
reason why the state machine regarding the valid VM's state transition
needed to be updated accordingly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
To follow a symmetrical model, and avoid potential race conditions, it's
important to restore a previously snapshot VM in a "paused" state.
The snapshot operation being valid only if the VM has been previously
paused.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
When the hypervisor crate was introduced, a few places that handled
errors were commented out in favor of unwrap, but that's bad practice.
Restore proper error handling in those places in this patch.
We cannot use from_raw_os_error anymore because it is wrapped deep under
hypervisor crate. Create new custom errors instead.
Fixes: e4dee57e81 ("arch, pci, vmm: Initial switch to the hypervisor crate")
Signed-off-by: Wei Liu <liuwe@microsoft.com>
This commit fixes some warnings introduced in the previous
hyperviosr crate PR.Removed some unused variables from arch/aarch64
module.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Start moving the vmm, arch and pci crates to being hypervisor agnostic
by using the hypervisor trait and abstractions. This is not a complete
switch and there are still some remaining KVM dependencies.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Implement the vCPU state getter and setter separately from the initial
KVM Hypervisor trait implementation, mostly for readability purposes.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
For each of the traits we are defining kvm related structures
and add the trait implementation to the structs. For more information
please see the kvm-ioctls and kvm-bindings crate.
This is a standalone implementation that does not include the switch of
the Cloud-Hypervisor vmm and arch crates to it.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
As the only hypervisor that Cloud-Hypervisor supports is KVM, the
Hypervisor trait accomodates for the upcoming KVM implementation.
This trait will be instanciated at build time through hypervisor
specific features, i.e. it's not aiming at run-time selection of
hypervisors for Cloud-Hypervisor.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
This VM trait should be implemented by each underlying hypervisor.
Previously created hypervisor object should create the VM based on
already selected hypervisor. This is just the trait definition. For each
of supported hypervisor we need to implement the trait. Later we will
implement this trait for KVM and then Microsoft Hyper-V.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
This Vcpu trait should be implemented by each underlying hypervisor.
Previously created hypervisor object should create the VM based on
already selected hypervisor and Vm object should create this vcpu
object based on same hyperviosr. Each of this object should be
referenced by trait object i.e <dyn Vcpu>.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The purpose of this trait is to add support for other hypervisors than
KVM, like e.g. Microsoft Hyper-V.
Further commits will define additional hypervisor related traits like
Vcpu and Vm. Each of the supported hypervisor will need to implement all
traits defined from the hypervisor crate.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Add multiple integration tests with various different CPU topologies and
check that they work as expected.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
There are two CPUID leaves for handling CPU topology, 0xb and 0x1f. The
difference between the two is that the 0x1f leaf (Extended Topology
Leaf) supports exposing multiple die packages.
Fixes: #1284
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The extended topology leaf (0x1f) also needs to have the APIC ID (which
is the KVM cpu ID) set. This mirrors the APIC ID set on the 0xb topology
leaf
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Rather than saving the individual parts into the CpuManager save the
full struct as it now also contains the topology data.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This needed to be updated to include specifying the boot and maxmium
vCPUs as well as the newly added topology for those vCPUs.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This allows the user to optionally specify the desired CPU topology. All
parts of the topology must be specified and the product of all parts
must match the maximum vCPUs.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This patch has been cherry-picked from the Firecracker tree. The
reference commit is 78ca0a942f32140465c67ea4b45d68c52c72d751.
Signed-off-by: Gabriel Ionescu <gbi@amazon.com>
Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
This patch has been cherry-picked from the Firecracker tree. The
reference commit is 6dbe8e021a64ba3742081741a7538cdfd93a102e.
Signed-off-by: Gabriel Ionescu <gbi@amazon.com>
Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
This patch has been cherry-picked from the Firecracker tree. The
reference commit is 78819f35f63f5777a58e3e1e774b3270b32881ed.
The vsock TX buffer flush operation would report inconsistent results,
under specific circumstances.
The flush operation is performed in two steps, since it's dealing with a
ring buffer, an the data to be flushed may wrap around. If the first
step was successful, but the second one failed, the whole flush
operation would report an error, thus causing flow control accounting to
lose track of the bytes that were successfully written by the first
pass.
This commit changes the flush behavior to always report success when
some data has been written to the backing stream.
Signed-off-by: Dan Horobeanu <dhr@amazon.com>
Signed-off-by: Gabriel Ionescu <gbi@amazon.com>
Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
This patch has been cherry-picked from the Firecracker tree. The
reference commit is 2da612a9cdce85c91fb54ab22d950ec6ccc93b27.
Fixed a bug introduced by a271d08f0b1ba0ee82761cd49244b6a8017bcede,
whereby the flow control accouting would be off by a few bytes, for
host-initiated connections.
The connection ack message ("OK <port_num><CR>") was accounted for as
data sent by the guest, so its length was substracted from the total
amount of data the guest was allowed to send.
This commit changes the way this ack message is sent, so that it
bypasses flow control accouting.
Signed-off-by: Dan Horobeanu <dhr@amazon.com>
Signed-off-by: Gabriel Ionescu <gbi@amazon.com>
Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
This patch has been cherry-picked from the Firecracker tree. The
reference commit is 109e631566350867dafa4b16c3919dfd1533eeea.
This commit changes the vsock connection state machine behavior to absorb
any EWOULDBLOCK errors recevied while handling an EPOLLOUT event. Previously,
this condition would lead to immediate connection termination.
Signed-off-by: Dan Horobeanu <dhr@amazon.com>
Signed-off-by: Gabriel Ionescu <gbi@amazon.com>
Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
This patch has been cherry-picked from the Firecracker tree. The
reference commit is 660d18cf7fee5b38c3b1b17a5da6544b9025909d.
Apparently, epoll_wait sometimes yields false EPOLLIN events (i.e. events
follwing which read() would fail with EWOULDBLOCK). This would cause the
vsock connection state machine to terminate connections, since an error
was detected on the underlying Unix socket.
This commit changes the vsock connection state machine code to handle such
erroneous EPOLLIN events by absorbing EWOULDBLOCK read() errors.
Signed-off-by: Dan Horobeanu <dhr@amazon.com>
Signed-off-by: Gabriel Ionescu <gbi@amazon.com>
Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
This patch has been cherry-picked from the Firecracker tree. The
reference commit is 1cc8b8a678eb28b20f5843556bdb7fbb2dfa6284.
Fixed a logical error in the vsock flow control, that would cause credit
update packets to not be sent at the right time.
Signed-off-by: Dan Horobeanu <dhr@amazon.com>
Signed-off-by: Gabriel Ionescu <gbi@amazon.com>
Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
This patch has been cherry-picked from the Firecracker tree. The
reference commit is d2475773557c82d2abad2fc8bdf69e7d01444109.
Fixed a vsock muxer issue that would cause a connection to be removed
from the RX queue, even though it still had pending RX data.
Signed-off-by: Dan Horobeanu <dhr@amazon.com>
Signed-off-by: Gabriel Ionescu <gbi@amazon.com>
Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
This patch adds `is_empty` method to VsockPacket to fix the
following clippy error:
error: item `vsock::packet::VsockPacket` has a public `len` method but no corresponding `is_empty` method
--> vm-virtio/src/vsock/packet.rs:100:1
|
100 | / impl VsockPacket {
101 | | /// Create the packet wrapper from a TX virtq chain head.
102 | | ///
103 | | /// The chain head is expected to hold valid packet header data. A following packet buffer
... |
334 | | }
335 | | }
| |_^
|
= note: `-D clippy::len-without-is-empty` implied by `-D warnings`
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#len_without_is_empty
Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
Currently, not every feature of the cloud-hypervisor is enabled
on AArch64, which means that on AArch64 machines, the
`run_unit_tests.sh` needs to be tailored and some unit test cases
should be run on x86_64 only.
Also this commit fixes the typo and unifies `Arm64` and `AArch64`
in the AArch64 document.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Its test case calls remove unconditionally. Instead of making the test
code call remove conditionally, removing the pci_support dependency
simplifies things -- that function is just a wrapper around HashMap's
remove function anyway.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
cloud-hypervisor: 763.978581807s: ERROR:pci/src/vfio.rs:651 -- failed to remove all guest memory regions from iommu table
when poweroff a vm with vfio device, clh will finally remove all guest memory region from iommu table
with the method unset_dma_map, not method setup_dma_map.
Signed-off-by: LiYa'nan <oliverliyn@gmail.com>
Upon PCI hotplug, the VMM now returns some information about the device
name and the associated b/d/f. This patch extends the integration tests
so that we validate the response is the one that is expected.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that PCI device hotplug returns a response, the OpenAPI definition
must reflect it, describing what is expected to be received.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to provide a more comprehensive b/d/f to the user, the
serialization of PciDeviceInfo is implemented manually to control the
formatting.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This patch completes the series by connecting the dots between the HTTP
frontend and the device manager backend.
Any request to hotplug a VFIO, disk, fs, pmem, net, or vsock device will
now return a response including the device name and the place of the
device in the PCI topology.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Pass from the device manager to the calling code the information about
the PCI device that has just been hotplugged.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to provide the device name and PCI b/d/f associated with a
freshly hotplugged device, the hotplugging functions from the device
manager return a new structure called PciDeviceInfo.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The support of AArch64 is in very early stage. The steps in building and
runing on X86 and AArch64 can not align well yet. Adding AArch64 content
to README.md would produce much divergence.
Adding a guide in docs/ folder could be a better way to start now.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
X86 and AArch64 work in different ways to shutdown a VM.
X86 exit VMM event loop through ACPI device;
AArch64 need to exit from CPU loop of a SystemEvent.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Screened IO bus because it is not for AArch64.
Enabled Serial, RTC and Virtio devices with MMIO transport option.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Implemented GSI allocator and system allocator for AArch64.
Renamed some layout definitions to align more code between architectures.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
This fixes the build warning below
warning: use of deprecated item 'tap::tests::pnet::datalink::NetworkInterface::mac_address': Please use NetworkInterface's field 'mac' instead.
--> net_util/src/tap.rs:530:24
|
530 | (interface.mac_address(), tx, rx)
| ^^^^^^^^^^^
|
= note: `#[warn(deprecated)]` on by default
Signed-off-by: Anatol Belski <ab@php.net>
For correctness, when the CPUID supports the LA57 feature, the VMM sets
the CR4.LA57 register, which means a fifth level of page table might be
needed. Even if it's not needed because the kernel should not use
addresses over 1GiB, it's better to define this new level anyway.
This patch only applies to the Linux boot codepath, which means it
affects both vmlinux without PVH and bzImage binaries. The bzImage
does not need this since the page tables and CR4 registers are set in
the decompression code from the kernel.
And for vmlinux with PVH, if we follow the PVH specification, the kernel
must be responsible for setting things up, but the implementation is
missing. This means for now that PVH does not support LA57 with 5 levels
of paging.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
"cc" is invoked as part of the Cloud Hypervisor Rust build however due
to a copy and paste error the wrong variable was being tested for
overriding the CC and the CFLAGS.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This revised version of the patch reuses the back-off strategy from
'ssh_command()' to deal with varying booting time.
Fixes: #1209
Signed-off-by: Bo Chen <chen.bo@intel.com>
Some CPUs might support up to 52 bits of addressable space, that's the
reason why we need the guest kernel to support 5 levels of page table.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to workaround a Linux bug that happens when we place devices at
the end of the physical address space on recent hardware (52 bits limit)
we reduce the MMIO address space by one 4k page. This way, nothing gets
allocated in the last 4k of the address space, which is negligible given
the amount of space in the address space.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In case the host CPU exposes the support for LA57 feature through its
cpuid, the CR4.LA57 bit is enabled accordingly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Currently released vm-memory uses aligned and volatile copying for all
data. The version in the fork only uses the assured (and slower) path
for data upto the natural data width.
Fixes: #1258
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This commit adds the implementation for the AArch64 PL031
Real Time Clock (RTC) that provides a long time base counter. This
is achieved by generating an interrupt signal after counting a
programmed number of cycles of a real-time clock input. The AArch64
guest VM of the cloud-hypervisor will use this RTC to sync the time
in itself.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
The action of "vm.delete" should not report errors on non-booted
VMs. This patch also revised the "docs/api.md" to reflect the right
'Prerequisites' of different API actions, e.g. on "vm.delete" and
"vm.boot".
Fixes: #1110
Signed-off-by: Bo Chen <chen.bo@intel.com>
This removes the need to use CAP_NET_ADMIN privileges and instead the
host MAC addres is either provided by the user or alternatively it is
retrieved from the kernel.
TEST=Run cloud-hypervisor without CAP_NET_ADMIN permission and a
preconfigured tap device:
sudo ip tuntap add name tap0 mode tap
sudo ifconfig tap0 192.168.249.1 netmask 255.255.255.0 up
cargo clean
cargo build
target/debug/cloud-hypervisor --serial tty --console off --kernel ~/src/rust-hypervisor-firmware/target/target/release/hypervisor-fw --disk path=~/workloads/clear-33190-kvm.img --net tap=tap0
VM was also rebooted to check that works correctly.
Fixes: #1274
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This can be used to preserve the host MAC address as part of the
configuration when the TAP device is precreated.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This allows an existing TAP interface to be used without needing
CAP_NET_ADMIN permissions on the Cloud Hypervisor binary as the ioctl to
bring up the interface is avoided.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
If the server returns an error then print out the response body if there
is one present.
Fixes: #1262
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In AArch64 prototype, there are code to construct the flattened device
tree, and to make such code compilable we need to install libfdt-dev. In
normal situation, this installation process can be done by either installing
libfdt-dev locally or in the development container.
Before formal AArch64 CI is setup, we use the workaround in this commit
to install libfdt in the github cross-build workflow.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
When booting VM on AArch64 machines, we need to construct the
flattened device tree before loading kernel. Hence here we add
the implementation of the flattened device tree for AArch64.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
As on AArch64 systems we need register mpidr to create the
flattened device tree, here in this commit we add ported AArch64
register implementation from Firecracker and related changes to
make this commit build.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This commit adds ported code of Generic Interrupt Controller (GIC)
software implementation for AArch64, including both GICv2 and
GICv3 devices. These GIC devices are actually emulated by the
host kernel through KVM and will be used in the guest VM as the
interrupt controller for AArch64.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This commit adds the memory layout design for AArch64 in
`arch/src/aarch64/layout.rs` and related changes in
`arch/src/lib.rs` to make sure this commit can build.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
The API has change to use generic GuestMemory trait:
pub fn get_host_address_range<M: GuestMemory>(
mem: &M,
addr: GuestAddress,
size: usize,
) -> Option<*mut u8> {
Signed-off-by: Arron Wang <arron.wang@intel.com>
There is a much stronger PCI dependency from vfio_pci.rs than a VFIO one
from pci/src/vfio.rs. It seems more natural to have the PCI specific
VFIO implementation in the PCI crate rather than the other way around.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Now the flow of both architectures are aligned to:
1. load kernel
2. create VCPU's
3. configure system
4. start VCPU's
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Between X86 and AArch64, there is some difference in booting a VM:
- X86_64 can setup IOAPIC before creating any VCPU.
- AArch64 have to create VCPU's before creating GIC.
The old process is:
1. load_kernel()
load kernel binary
configure system
2. activate_vcpus()
create & start VCPU's
So we need to separate "activate_vcpus" into "create_vcpus" and
"activate_vcpus" (to start vcpus only). Setup GIC and create FDT
between the 2 steps.
The new procedure is:
1. load_kernel()
load kernel binary
(X86_64) configure system
2. create VCPU's
3. (AArch64) setup GIC
4. (AArch64) configure system
5. start VCPU's
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
The general handling of VIRTIO_RING_F_EVENT_IDX is in the
vhost_user_backend functionality and the net specific handling is in the
NetQueuePair from virtio-net.
As such enabling for the vhost-user-net backend is just the case of
adding the feature.
Fixes: #789
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
If VIRTIO_RING_F_EVENT_IDX is negotiated only generate suppress
interrupts if the guest has asked us to do so.
Fixes: #788
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In some situations it is seen that the first interrupt sent to the guest
is lost upon a restore (due to the tap worker being awake ahead of the
vPUs).
This causes problems with VIRTIO_RING_F_EVENT_IDX interrupt suppression
as the guest will not be interrupted again in order to mitigate this we
always interrupt the guest until the device itself has been signalled by
the guest.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The logic for handling the networking queues can now be shared between
the version running in vhost-user-net and vm-virtio.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This requires exposing the struct members and also using Option<..>
types for the main epoll fd and the memory as they are initialised later
in vhost-user-net.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Split handling of behaviour that is independent of the device itself so
that it can be reused in the vhost-user-net device.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Split out functions that work just on the TAP device and queues. Whilst
doing so also improve the error handling to return Results rather than
drop errors.
This change also addresses a bug where the TAP event suppression could
ineffectual because it was being enabled immediately after it may have
been disabled:
resume_rx -> rx_single_frame -> unregister_listener -> resume_rx ->
register_listener.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Instead of responding only headers with error code, we now return
complete error responses to HTTP requests with errors (e.g. undefined
endpoints and InternalSeverError).
Fixes: #472
Signed-off-by: Bo Chen <chen.bo@intel.com>
When doing self spawning the child will attempt to set the umask() again. Let
it through the seccomp rules so long as it the safe mask again.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This commit only implements the InterruptController crate on AArch64.
The device specific part for GIC is to be added.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
IOAPIC, a X86 specific interrupt controller, is referenced by device
manager and CPU manager. To work with more architectures, a common
type for all architectures is needed.
This commit introduces trait InterruptController to provide architecture
agnostic functions. Device manager and CPU manager can use it without
caring what the underlying device is.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
The result of the workflow can be seen in Checks tab of a PR.
Two targets have been added:
- stable aarch64-unknown-linux-gnu
- stable aarch64-unknown-linux-musl
Note: a temporary step was added before building. We used "sed" command
to remove "with-serde" feature of kvm-bindings in vmm/Cargo.toml.
This step should be removed in future when kvm-bindings is ready.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
This is a preparing commit to build and test CH on AArch64. All building
issues were fixed, but no functionality was introduced.
For X86, the logic of code was not changed at all.
For ARM, the architecture specific part is still empty. And we applied
some tricks to workaround lint warnings. But such code will be replaced
later by other commits with real functionality.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Updated Dockerfile to work with multiple architectures.
Updated dev_cli.sh to:
1. Build container image before AArch64 image is ready in public.
2. Adjust default feature collection on AArch64.
3. Workaround a build problem with musl on AArch64.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
This config option provided very little value and instead we now enable
this feature (which then lets the guest control the cache mode)
unconditionally.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Correctly implement the virtio specification by setting the writeback
field on the request based on the algorithm in the spec.
TEST=Boot with hypervisor-firmware with CH in verbose mode. See info
level messages saying cache mode is writethrough in firmware (no support
for flush or WCE). Once in the Linux kernel see messages that mode is
writeback.
Fixes: #1216Fixes: #680
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Correctly implement the virtio specification by setting the writeback
field on the request based on the algorithm in the spec.
TEST=Boot with hypervisor-firmware with CH in verbose mode. See info
level messages saying cache mode is writethrough in firmware (no support
for flush or WCE). Once in the Linux kernel see messages that mode is
writeback.
Fixes: #1216
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When this is set to false the write needs to be followed by a flush on
the underlying disk (leading to a fsync()).
The default behaviour is not changed with this change.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Previous to adding a a trait method to inform the backends of the acked
features backends can use features than the guest has not enabled which
could lead to unpredictable results.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Move the method that is used to decide whether the guest should be
signalled into the Queue implementation from vm-virtio. This removes
duplicated code between vhost_user_backend and the vm-virtio block
implementation.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The implementation of this virtio block (and vhost-user block) command
called a function that was a no-op on Linux. Use the same function as
virtio-pmem to ensure that data is not lost when the guest asks for it
to be flused to disk.
Fixes: #399
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Now that snapshot/restore support has been enabled for virtio-vsock, the
corresponding integration test is expanded with some validation that
virtio-vsock supports to be snapshot and restored.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
When compiled with pci feature, the integration test now validates that
/dev/vdb can be correctly read while being placed behing a virtual
IOMMU.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
For both virtio-mmio and virtio-pci transport layers, we were setting
every field from the saved snapshot during a restore. This is a problem
when we don't want to override specific fields such as iommu_mapping_cb
because the saved snapshot doesn't contain the appropriate information.
That's why this commit sets only the appropriate field from the saved
snapshot during a restore.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Provide implementation for both snapshot() and restore() methods from
the Snapshottable trait, so that we can snapshot and restore a VM with
devices attached to a virtual IOMMU.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The integration test validating that --serial off works correctly was
not properly written as it was using the FW, which by default would use
the kernel command line found in the EFI partition. Unfortunately, this
kernel command line was including "console=ttyS0", which causes the
kernel to try to write to the serial port, even if there's no serial
port being emulated.
The problem is, when no emulation of the serial port is provided, the
default value returned on 0x3f8 is 0, which makes the guest kernel think
that some data needs to be read.
The only way to avoid all this is by ensuring we can control the kernel
command line by removing any occurence of "console=ttyS0" from it.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Explicit call to 'close()' is required on file descriptors allocated
from 'epoll::create()', which is missing for the 'EpollContext' and
'VringWorker'. This patch enforces to close the file descriptors by
reusing the Drop trait of the 'File' struct.
Signed-off-by: Bo Chen <chen.bo@intel.com>
According to the virtio spec the guest should always be interrupted when
"used" descriptors are returned from the device to the driver. However
this was not the case for the TX queue in either the virtio-net
implementation or the vhost-user-net implementation.
This would have meant that the guest could end up with a reduced TX
throughput as it would not know that the packets had been dispatched via
the VMM.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
As the parsing code is reused the flush feature is already implemented
and ready to be used.
Fixes: #1197
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add missing syscall used by the musl build.
TEST=scripts/dev_cli.sh tests --libc musl --integration -- vhost_user_fs_daemon
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Extend the set of tests we have for virtio-net and vhost-user-net to
check for host MAC address setting.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add a new "host_mac" parameter to "--net" and "--net-backend" and use
this to set the MAC address on the tap interface. If no address is given
one is randomly assigned and is stored in the config.
Support for vhost-user-net self spawning was also included.
Fixes: #1177
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Do this by reading the HW address information and then modifying the
HW address to match the desired address. Preserving the rest of the
state including the address type.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The clock_nanosleep system call needs to be whitelisted since the commit
12e00c0f45 introduced the use of a sleep()
function. Without this patch, we can see an error when the VM is paused
or killed.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Implement seccomp; we use one filter for all threads.
The syscall list comes from the C daemon with syscalls added
as I hit them.
The default behaviour is to kill the process, this normally gets
audit logged.
--seccomp none disables seccomp
log Just logs violations but doesn't stop it
trap causes a signal to be be sent that can be trapped.
If you suspect you're hitting a seccomp action then you can
check the audit log; you could also switch to running with 'log'
to collect a bunch of calls to report.
To see where the syscalls are coming from use 'trap' with a debugger
or coredump to backtrace it.
This can be improved for some syscalls to restrict the parameters
to some syscalls to make them more restrictive.
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Like the actions that don't take data such as "pause" or "resume" use a
common handler implementation to remove duplicated code for handling
simple endpoints like the hotplug ones.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Many of the API requests take a similar form with a single data item
(i.e. config for a device hotplug) expand the VmAction enum to handle
those actions and a single function to dispatch those API events.
For now port the existing helper functions to use this new API. In the
future the HTTP layer can create the VmAction directly avoiding the
extra layer of indirection.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Rather than save the save a function pointer and use that instead the
underlying action. This is useful for two reasons:
1. We can ensure that we generate HttpErrors in the same way as the
other endpoints where API error variant should be determined by the
request being made not the underlying error.
2. It can be extended to handle other generic actions where the function
prototype differs slightly.
As result of this refactoring it was found that the "vm.delete" endpoint
was not connected so address that issue.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Extend the EndpointHandler trait to include automatic support for
handling PUT requests. This will allow the removal of lots of duplicated
code in the following commit from the API handling code.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Implement support for setting up a sandbox for running the
service. The technique for this has been borrowed from virtiofsd, and
consists on switching to new PID, mount and network namespaces, and
then switching root to the directory to be shared.
Future patches will implement additional hardening features like
dropping capabilities and seccomp filters.
Signed-off-by: Sergio Lopez <slp@redhat.com>
Changes is vhost crate require VhostUserDaemon users to create and
provide a vhost::Listener in advance. This allows us to adopt
sandboxing strategies in the future, by being able to create the UNIX
socket before switching to a restricted namespace.
Update also the reference to vhost crate in Cargo.lock to point to the
latest commit from the dragonball branch.
Signed-off-by: Sergio Lopez <slp@redhat.com>
Allow callers to provide a file descriptor for /proc/self/fd. This is
useful for sandboxing, as we may be running in a namespace that
doesn't have access to /proc.
Signed-off-by: Sergio Lopez <slp@redhat.com>
Open a file descriptor to /proc/self/fd instead of /proc. We aren't
using any other entries from that directory, and doing this allows us
to keep working even if /proc is no longer present in our
namespace (useful for sandboxing).
Signed-off-by: Sergio Lopez <slp@redhat.com>
There's a simple way to trigger PCI BAR reprogramming for a given
device, by removing it and then hotplugging it back. The Linux kernel
will simply choose to place the BARs at different location than the ones
chosen by Cloud-Hypervisor. By doing so, and creating the snapshot after
this hotplug operation, we can manage to validate that the resource are
correctly restored for a given virtio-pci device.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that snapshot/restore feature is stable for both virtio-mmio and
virtio-pci devices, we re-enable the integration test for validating
snapshot/restore does not get broken.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The ch branch has been rebased to incorporate the latest upstream code
requiring a small change to the unit tests.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Now that virtio-fs has unplug support try unplugging and readding the
virtio-fs device.
Fixes: #1050
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
By passing a reference of the DeviceTree to the AddressManager, we can
now update the DeviceTree whenever a PCI BAR is reprogrammed. This is
mandatory to maintain the correct resources information related to each
virtio-pci device, which will ensure correct information will be stored
upon VM snapshot.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We want to be able to share the same DeviceTree across multiple threads,
particularly to handle the use case where PCI BAR reprogramming might
need to update the tree while from another thread a new device is being
added to the tree.
That's why this patch moves the DeviceTree instance into an Arc<Mutex<>>
so that we can later share a reference of the same mutable tree with the
AddressManager responsible for handling PCI BAR reprogramming.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By using the vector of resources provided by the DeviceNode, the device
manager can store the information related to PCI BARs from a virtio-pci
device. Based on this, and upon VM restoration, the device manager can
restore the BARs in the expected location in the guest address space.
One thing to note is that we only need to provide the VirtioPciDevice
with the configuration BAR (BAR 0) since the SHaredMemory BAR info comes
from the virtio device directly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
From a VirtioPciDevice perspective, there are two types of BARs, either
the virtio configuration BAR or the SHaredMemory BAR.
The SHaredMemory BAR address comes from the virtio device directly as
the memory region had been previously allocated when the virtio device
has been created. So for this BAR, there's nothing to do when restoring
a VM, since the associated virtio device is already restored with the
appropriate resources, hence the BAR will already be at the right
address.
The remaining configuration BAR is different, as we usually get its
address from the SystemAllocator. This means in case we restore a VM,
we must provide this value, bypassing the allocator. This is what this
commit takes care of, by letting the caller set the base address for the
configuration BAR prior to allocating the BARs.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the new field "pci_bdf", a virtio-pci device can be restored at
the same place on the PCI bus it was located before the VM snapshot.
This ensures consistent placement on the PCI bus, based on the stored
information related to each device.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We need a way to store the information about where a PCI device was
placed on the PCI bus before the VM was snapshotted. The way to do this
is by adding an extra field to the DeviceNode structure.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to let the PciBus user choose where a device should be placed
on the bus, a new function get_device_id() is introduced. This will be
helpful in the context of snapshot/restore as the caller will be able to
place the PCI devices on the same slot they were placed before the
snapshot was taken.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Use the ACPI feature to control whether to build the mptable. This is
necessary as the mptable and ACPI RSDP table can easily overwrite each
other leading to it failing to boot.
TEST=Compile with default features and see that --cpus boot=48 now
works, try with --no-default-features --features "pci" and observe the
--cpus boot=48 also continues to work.
Fixes: #1132
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The mmio integration test build currently doesn't use ACPI so piggyback
on that test variation.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The setup_mptables() call which is not used on ACPI builds has a side
effect of testing whether there was enough RAM which one of the unit
tests was relying on. Add a similar check for the RSDP address.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This gives Cloud-Hypervisor the possibility to snapshot and restore a
VM running with virtio-pci devices attached to it.
The VirtioPciDevice snapshot contains a vector of sub-snapshots to store
and restore information related to MsixConfig, VirtioPciCommonConfig and
PciConfiguration structures, along with snapshot data related to
VirtioPciDevice itself.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This structure contains all the virtio generic information, and as part
of restoring a VM with virtio-pci devices, it is important to restore
these values to ensure the device's proper functioning.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The PCI configuration from each PCI device is modified at runtime as we
can expect the guest OS to write to some PCI capability structure, or
move the BAR to a different location in the guest address space.
For all the reasons why such configuration might differ from the initial
configuration, we must store the registers values to be able to restore
them with the right values whenever a PCI device is being restored.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to restore devices relying on MSI-X, the MsixConfig structure
must be restored with the correct values. Additionally, the KVM routes
must be restored so that interrupts can be delivered through KVM the way
they were configured before the snapshot was taken.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Continue to try and track down the instability in the numbers generated
by this test by removing the (unused) network interface. It's possible
we are getting broadcast packets into the tap device and generating a
variable size of allocations.
See: #760
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Switch to using the recently added OptionParser in the code that parses
the block backend.
Fixes: #1092
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Switch to using the recently added OptionParser in the code that parses
the network backend.
Whilst doing this also update the net-backend syntax to use "sock"
rather than socket.
Fixes: #1092
Partially fixes: #1091
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Rather than repeat syntax for the vhost-user-block backend in multiple
places store it in one place and reference it from the required places.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Rather than repeat syntax for the vhost-user-net backend in multiple
places store it in one place and reference it from the required places.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Upon a virtio reset, the driver expects that available and used indexes
will be reset to 0. That's why we need to reset these values from the
VMM for any virtio device that might get reset.
This issue was not detected before because the Vec<Queue> maintained
through VirtioPciDevice or MmioDevice was never updated from the virtio
device thread after the device had been actived. For this reason, upon
reset, both available and used indexes were already at the value 0.
The issue arose when trying to reset a device after the VM was restored.
That's because during the restore, each queue is assigned with the right
available and used indexes before it is passed to the device through the
activate function. And that's why upon reset, each queue was still
assigned with these indexes while it should have been reset to 0.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
All our virtio devices support to be reset, but the virtio-mmio
transport layer was not implemented for it. This patch fixes this
lack of support.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The ch-remote usage says:
OPTIONS:
--api-socket <api-socket> HTTP API socket path (UNIX domain socket).
However it doesn't seem to actually accept that syntax, instead
requiring "--api-socket=". This may be a clap bug however it is resolved
by setting the number of arguments requires to exactly one. Which is
also the actual correct number.
Fixes: #1117Fixes: #1116
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
To aid the debugging of why the test_memory_overhead sometimes fails
print a details of all the named regions that it uses in its
calculation.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The current tests support giving a tap name but they only test with tap
interfaces that already exist.
Fixes: #871
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Stripping the release build for glibc shrinks the size considerably:
$ du -h target/release/cloud-hypervisor
8.5M target/release/cloud-hypervisor
$ strip target/release/cloud-hypervisor
$ du -h target/release/cloud-hypervisor
5.2M target/release/cloud-hypervisor
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
To avoid a race condition where the signal might "miss" the KVM_RUN
ioctl() instead reapeatedly try sending a signal until the vCPU run is
interrupted (as indicated by setting a new per vCPU atomic.)
It important to also clear this atomic when coming out of a paused
state.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
To ensure that the DeviceManager threads (such as those used for virtio
devices) are cleaned up it is necessary to unpark them so that they get
cleanly terminated as part of the shutdown.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
After setting the kill signal flag for the vCPU thread release the pause
flag and unpark the threads. This ensures that that the vCPU thread will
wake up and check the kill signal flag if the VM is paused.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Rather than immediately entering the vCPU run() code check if the kill
signal is set. This allows paused VMs to be shutdown.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This module will be dedicated to DeviceNode and DeviceTree definitions
along with some dedicated unit tests.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This iterator will let the VMM enumerate the resources associated
with the DeviceManager, allowing for introspection.
Moreover, by implementing a double ended iterator, we can get the
hierarchy from the leaves to the root of the tree, which is very
helpful in the context of restoring the devices in the right order.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that the device tree fully replaced the need for a dedicated list of
migratable devices, this commit cleans up the codebase by removing it
from the DeviceManager.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit switches from migratable_devices to device_tree in order to
restore devices exclusively based on the device tree.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit adds an extra field to the DeviceNode so that the structure
can hold a Migratable device. The long term plan is to be able to remove
the dedicated table of migratable devices, but instead rely only on the
device tree.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to hide the complexity chosen for the device tree stored in
the DeviceManager, we introduce a new DeviceTree structure.
For now, this structure is a simple passthrough of a HashMap, but it can
be extended to handle some DeviceTree specific operations.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This device has a dedicated memory region in the guest address space,
which means in case of snapshot/restore, it must be restored in the
exact same location it was during the snapshot.
That's through the resources that we can describe the location of this
extra memory region, allowing the device for correct restoring.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This device has a dedicated memory region in the guest address space,
which means in case of snapshot/restore, it must be restored in the
exact same location it was during the snapshot.
That's through the resources that we can describe the location of this
extra memory region, allowing the device for correct restoring.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the device tree, retrieve the resources associated with a
virtio-mmio device to restore it at the right location in guest address
space. Also, the IRQ number is correctly restored.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Instead of splitting the MMIO allocation and the device creation into
separate functions for virtio-mmio devices, it's is easier to move
everything into the same function as we'll be able to gather resources
in the same place for the same device.
These resources will be stored in the device tree in a follow up patch.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In case the VM is created from scratch, the devices should be created
after the DeviceManager has been created. But this should not affect the
restore codepath, as in this case the devices should be created as part
of the restore() function.
It's necessary to perform this differentiation as the restore must go
through the following steps:
- Create the DeviceManager
- Restore the DeviceManager with the right state
- Create the devices based on the restored DeviceManager's device tree
- Restore each device based on the restored DeviceManager's device tree
That's why this patch leverages the recent split of the DeviceManager's
creation to achieve what's needed.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit performs the split of the DeviceManager's creation into two
separate functions by moving anything related to device's creation after
the DeviceManager structure has been initialized.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the device tree, we now ensure the restore can be done in the
right order, as it will respect the dependencies between nodes.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The DeviceManager itself must be snapshotted in order to store the
information regarding the devices associated with it, which effectively
means we need to store the device tree.
The mechanics to snapshot and restore the DeviceManagerState are added
to the existing snapshot() and restore() implementations.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The DeviceManager now creates a tree of devices in order to store the
resources associated with each device, but also to track dependencies
between devices.
This is a key part for proper introspection, but also to support
snapshot and restore correctly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on rust-vmm code, we copy the Resource structure as it will be
used to define the set of resources associated with a device.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
It's not possible to call UnixListener::Bind() on an existing file so
unlink the created socket when shutting down the Vsock device.
This will allow the VM to be rebooted with a vsock device.
Fixes: #1083
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Generate an error during validation if an attempt it made to place a
device behind an IOMMU or using a VFIO device when not using PCI.
Fixes: #751
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Let's put an underscore "_" in front of each device name to identify
when it has been set internally.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The virtio-console was not added to the list of Migratable devices,
which is fixed from this patch.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This identifier is chosen from the DeviceManager so that it will manage
all identifiers across the VM, which will ensure uniqueness.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This identifier is chosen from the DeviceManager so that it will manage
all identifiers across the VM, which will ensure uniqueness.
It is based off the name from the virtio device attached to this
transport layer.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This identifier is chosen from the DeviceManager so that it will manage
all identifiers across the VM, which will ensure uniqueness.
It is based off the name from the virtio device attached to this
transport layer.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Because we know we will need every virtio device to be identified with a
unique id, we can simplify the code by making the identifier mandatory.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This identifier is chosen from the DeviceManager so that it will manage
all identifiers across the VM, which will ensure uniqueness.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This identifier is chosen from the DeviceManager so that it will manage
all identifiers across the VM, which will ensure uniqueness.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This identifier is chosen from the DeviceManager so that it will manage
all identifiers across the VM, which will ensure uniqueness.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This identifier is chosen from the DeviceManager so that it will manage
all identifiers across the VM, which will ensure uniqueness.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This identifier is chosen from the DeviceManager so that it will manage
all identifiers across the VM, which will ensure uniqueness.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This identifier is chosen from the DeviceManager so that it will manage
all identifiers across the VM, which will ensure uniqueness.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This identifier is chosen from the DeviceManager so that it will manage
all identifiers across the VM, which will ensure uniqueness.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This identifier is chosen from the DeviceManager so that it will manage
all identifiers across the VM, which will ensure uniqueness.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This identifier is chosen from the DeviceManager so that it will manage
all identifiers across the VM, which will ensure uniqueness.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This identifier is chosen from the DeviceManager so that it will manage
all identifiers across the VM, which will ensure uniqueness.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This identifier is chosen from the DeviceManager so that it will manage
all identifiers across the VM, which will ensure uniqueness.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This identifier is chosen from the DeviceManager so that it will manage
all identifiers across the VM, which will ensure uniqueness.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Even in the context of "mmio" feature, we need the next device name to
be generated as we need to identify virtio-mmio devices to support
snapshot and restore functionalities.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This will be later used to identify each device used by the VM in order
to perform introspection and snapshot/restore properly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This will be later used to identify each device used by the VM in order
to perform introspection and snapshot/restore properly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This will be later used to identify each device used by the VM in order
to perform introspection and snapshot/restore properly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This will be later used to identify each device used by the VM in order
to perform introspection and snapshot/restore properly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This will be later used to identify each device used by the VM in order
to perform introspection and snapshot/restore properly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
If the virtio-console device is supposed to be placed behind the virtual
IOMMU, this must be explicitly propagated through the code.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
If the virtio-rng device is supposed to be placed behind the virtual
IOMMU, this must be explicitly propagated through the code.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
If the virtio-vsock device is supposed to be placed behind the virtual
IOMMU, this must be explicitly propagated through the code.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We pass it to the integration and unit tests script through --libc.
Cargo tests are left unmodified.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Now that all virtio devices are assigned with identifiers, they could
all be removed from the VM. This is not something that we want to allow
because it does not make sense for some devices. That's why based on the
device type, we remove the device or we return an error to the user.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The parameters regarding the attachment to the virtio-iommu device was
not propagated correclty, and any modification to the configuration was
not stored back into it.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
It's possible to have multiple vsock devices so in preparation for
hotplug/unplug it is important to be able to have a unique identifier
for each device.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
If the current state is paused that means most of the handles got killed by pthread_kill
We need to unpark those threads to make the shutdown worked. Otherwise
The shutdown API hangs and the API is not responding afterwards. So
before the shutdown call we need to resume the VM make it succeed.
Fixes: #817
Signed-off-by: Muminul Islam <muislam@microsoft.com>
If a size is specified use it (in particular this is required if the
destination is a directory) otherwise seek in the file to get the size
of the file.
Add a new check that the size is a multiple of 2MiB otherwise the kernel
will reject it.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Check that if any device using vhost-user (net & disk with
vhost_user=true) or virtio-fs is enabled then check shared memory is
also enabled.
Fixes: #848
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The new 'shared' and 'hugepages' controls aim to replace the 'file'
option in MemoryConfig. This patch also updated all related integration
tests to use the new controls (instead of providing explicit paths to
"/dev/shm" or "/dev/hugepages").
Fixes: #1011
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Signed-off-by: Bo Chen <chen.bo@intel.com>
FS_IO is part of the actions a vhost-user-fs daemon can ask the VMM to
perform on its behalf. It is meant to read/write the content from a file
descriptor directly into a guest memory region. This region can either
be a RAM region or the dedicated cache region for virtio-fs.
The way FS_IO was implemented, it was only expecting the guest physical
address provided through the "cache_offset" field to refer to the cache
region. Unfortunately, this was only implementing FS_IO partially.
This patch extends the existing FS_IO implementation by checking the GPA
against the cache region as a first step, but if it is not part of the
cache region address range, then we fallback onto searching for a RAM
region that could match. If there is a matching RAM region, we retrieve
the corresponding host address to let the VMM read/write from/to it.
Fixes: #1054
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
The newly added integration test "test_snapshot_restore" is very
unstable, which causes our CI to fail on most pull requests, which is
not acceptable. That's why we ignore this test until we can fix the
stability issue.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that we have multiple virtio devices supporting snapshot and restore
operations, we can add a new integration test to validate the migration
feature works as expected.
The important part is virtio-net as it is used to ssh into the VM to
verify the VM has been restored in the proper state.
This test only works for virtio-mmio for now. Further support for
testing virtio-pci will be added once the virtio-pci transport layer
will support migration.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This doc describe how to use vhost-user-blk in Cloud Hypervisor
with SPDK. With this doc, we can setup simple test environment on
Ubuntu release.
Signed-off-by: Yang Zhong <yang.zhong@intel.com>
This patch implements the Snapshottable trait for virtio-console, which
enables migration support for it. A VM with a virtio-console device
attached can be snapshot and then restored without issues.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Replace alignment calculation of start address with functionally
equivalent version that does not assume that the block size is a power
of two.
Signed-off-by: Martin Xu <martin.xu@intel.com>
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The frame buffer must be updated depending on the amount read from it,
which depends on the number and depth of descriptors available at the
time of the processing.
This patch handles this buffer update, and allow for large buffers to be
correctly processed in multiple rounds.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
On the restore path, using the available and used indexes read from
memory to fill the Queue structure was a mistake. Indeed, the available
index is written from the guest and it reflects the last available index
in the descriptor table. But the driver might have queued a lot of
buffers which have not yet been used by the device. This leads to a
situation where the next_avail from Queue is completely different from
the one we can read from memory.
Instead, the right way to determine the next_avail index that should be
used by the device is by relying on the used index from the memory. This
index represents the correct information we're looking for as it has
been updated before the snapshot to let the guest know the next index to
process.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
First, this modifies the existing helpers on how to get indexes for
available and used rings from memory. Instead of updating the queue
through each helper, they are now used as simple getters.
Based on these new getters, we could create a new helper to determine if
the queue has some available descriptors already queued from the driver
side. This helper is going to be particularly helpful when trying to
determine from a virtio thread if a queue is already loaded with some
available buffers that can be used to send information to the guest.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
When a virtio device is dynamically removed from the VM through the
hot-unplug mechanism, every mapping associated with it must be properly
removed.
Based on the previous patches letting a VirtioDevice expose the list of
userspace mappings associated with it, this patch can now remove all the
KVM userspace memory regions through the MemoryManager.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The same way we added a helper for creating userspace memory mappings
from the MemoryManager, this patch adds a new helper to remove some
previously added mappings.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Since the virtio-fs device is backed by a vhost-user process, it is
important to implement the proper shutdown() function from the
VirtioDevice trait, as vhost-user-blk and vhost-user-net do.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
When hot-unplugging the virtio-pmem from the VM, we don't remove the
associated userspace mapping. This patch will let us fix this in a
following patch. For now, it simply adapts the code so that the Pmem
device knows about the mapping associated with it. By knowing about it,
it can expose it to the caller through the new userspace_mappings()
function.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This will help when we will implement the hot-unplug of the virtio-fs
device, as we will have to remove correctly the userspace mappings
associated with the device.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Introduce new getter function to the VirtioDevice trait, as it will
allow the caller to retrieve the list of userspace mappings associated
with the device.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
There is some duplication between regular and hotplug virtio-fs tests
that can be factorized by adding a simple hotplug flag to choose if each
test should run with or without hotplugging the device.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The hotplugged virtio devices were not added to the list of virtio
devices from the DeviceManager. This patch fixes it, as it was causing
hotplugged virtio-fs devices from not supporting memory hotplug, since
they were never getting the update as they were not part of the list of
virtio devices held by the DeviceManager.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Adds DeviceManager method `make_virtio_fs_device` which creates a single
device, and modifies `make_virtio_fs_devices` to use this method.
Implements the new `vm.add-fs route`.
Signed-off-by: Dean Sheather <dean@coder.com>
Currently unimplemented. Once implemented, this API will allow for
creating virtio-fs devices in the VM after it has booted.
Signed-off-by: Dean Sheather <dean@coder.com>
In the context of the shared memory region used by virtio-fs in order to
support DAX feature, the shared region is exposed as a dedicated PCI
BAR, and it is backed by a KVM userspace mapping.
Upon BAR remapping, the BAR is moved to a different location in the
guest address space, and the KVM mapping must be updated accordingly.
Additionally, we need the VirtioDevice to report the updated guest
address through the shared memory region returned by get_shm_regions().
That's why a new setter is added to the VirtioDevice trait, so that
after the mapping has been updated for KVM, we can tell the VirtioDevice
the new guest address the shared region is located at.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By adding the shared memory regions to the list of BARs, we make sure
the DeviceManager will register it as a BAR on the PCI bus. Without
this, when PCI BAR reprogramming happens, the PCI bus errors since it
does not know about any BAR at the specified address.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The order the elements are pushed into the list is important to restore
them in the right order. This is particularly important for MmioDevice
(or VirtioPciDevice) and their VirtioDevice counterpart.
A device must be fully ready before its associated transport layer
management can trigger its restoration, which will end up activating the
device in most cases.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Any virtio device relying on the mmio transport layer can be snapshotted
and restored thanks to this new patch. From the MmioDevice perspective,
it is mainly a matter of saving the information about the virtqueues as
the restore path will need them to activate the device (if needed
because it has been activated before being snapshotted).
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In anticipation for adding snapshot/restore support to virtio devices,
this commit introduces two new helpers updating the available and used
indexes of a queue, relying on the guest memory.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit relies on serde to serialize and deserialize the content of
a Queue structure. This will be useful information to store when
implementing snapshot/restore feature for virtio devices.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Add the WRITE_KILL_PRIV write flag, corresponding to
FUSE_WRITE_KILL_PRIV introduced in 7.31, and use to only remove the
setuid and setgid bits (by switching credentials) conditionally.
Signed-off-by: Sergio Lopez <slp@redhat.com>
Add support for MAX_PAGES, corresonding to FUSE_MAX_PAGES introduced
in FUSE 7.28.
This allows us to negotiate with the kernel the maximum number of
pages that we support to transfer in a single request.
Signed-off-by: Sergio Lopez <slp@redhat.com>
Add support for FOPEN_CACHE_DIR, a flag that allows us to tell the
guest that it's safe to cache a directory, introduced in FUSE 7.28.
Signed-off-by: Sergio Lopez <slp@redhat.com>
When reporting the BAR size it is necessary to return a value that is
encoded such that all the bits are set that represent the mask of the
natural alignment of the field.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add support for specifying the PCI revision in the PCI configuration and
populate this with the value of 1 for virtio-pci devices.
The virtio-pci specification is slightly ambiguous only saying that
transitional (i.e. devices that support legacy and virtio 1.0) should
set this to 0. In practice it seems that software expects the revision
to be set to 1 for modern only devices.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
OVMF and other standard firmwares use I/O port 0x402 as a simple debug
port by writing ASCII characters to it. This is gated under a feature
that is not enabled by default.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
After all the previous refactoring patches, we can finally create
multiple threads under the same backend. This is directly combined with
multiqueues so that it will create one thread per queue.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Anticipating the follow up patches to run multiple threads for the same
backend, we need the initialization of the disk to happen in the high
level structure VhostUserBlkBackend.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The DiskFile will need to be shared across multiple threads when running
multiple queues across these threads. That's why it needs to be put
inside an Arc. The reason for the Mutex is because execute() expects a
mutable object implementing Read + Write + Seek. Unfortunately, this
create a contention point as the object needs to be locked from each
thread, reducing the performance gain we will get with multiple threads.
The need for an immutable object would solve this problem, and it will
be addressed later through follow up patches.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
There is no need for retrieving the VringWorker since we don't need to
register some extra file descriptors to the epoll loop.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We now support assigning device ids for VFIO and virtio-pci devices so
this error can be generalised.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Extend the eject_device() method on DeviceManager to also support
virtio-pci devices being unplugged.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add an accessor to return the underlying VirtioDevice. This is useful
for managing the removal of the device from internal datastructures when
handling virtio-pci device unplug.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In order to support hotplugging there is a map of human readable device
id to PCI BDF map.
As the device id is part of the specific device configuration (e.g.
NetConfig) it is necessary to return the id through from the helper
functions that create the devices through to the functions that add
those devices to the bus. This necessitates changing a great deal of
function prototypes but otherwise has little impact.
Currently only if an id is supplied by the user as part of the device
configuration is it populated into this map. A later commit will
populate with an autogenerated name where none is supplied by the user.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In order to support freeing the memory that is allocated we need to make
sure that we update the internal representation so that free_bars() can
correctly free the memory if the device has its BARs moved.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Implement the free_bars() method from the PciDevice trait which is used
as part of the device removal process. Although there is only one BAR
allocated by VirtioPciDevice simplify the code by using a vector.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Now that ownership of the memory regions used for the virtio-pmem and
vhost-user-fs devices have been moved into those devices it is no longer
necessary to track them inside DeviceManager.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Move the release of the managed memory region from the DeviceManager to
the vhost-user-fs device. This ensures that the memory will be freed when
the device is unplugged which will lead to it being Drop()ed.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Move the release of the managed memory region from the DeviceManager to
the virtio-pmem device. This ensures that the memory will be freed when
the device is unplugged which will lead to it being Drop()ed.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
By implementing queues_per_thread(), this patch fills the last missing
bit to enable multithreaded multiqueue support for the vhost-user-net
backend implementation.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By adding a "thread_id" parameter to handle_event(), the backend crate
can now indicate to the backend implementation which thread triggered
the processing of some events.
This is applied to vhost-user-net backend and allows for simplifying a
lot the code since each thread is identical.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By adding the "thread_index" parameter to the function exit_event() from
the VhostUserBackend trait, the backend crate now has the ability to ask
the backend implementation about the exit event related to a specific
thread.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to prepare for the support of multithreaded multiqueues, the
structure VhostUserNetThread is simplified to hold only one RX queue,
one TX queue, and one TAP interface.
Following this change, VhostUserNetBackend now holds a list of threads
instead of going through each thread to handle multiqueues.
These changes decouple neatly the abstraction between the backend and
each thread. This allows for a lot of simplification since we now know
all threads are identical, hence the handling of events becomes very
straightforward.
One important point is that each thread can be locked when in use,
without causing any contention with other threads since the backend
doesn't need to be locked anymore.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that multiple worker threads can be run from the backend crate, it
is important that each backend implementation can access every worker
thread.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to support multiqueues running on multiple threads for
increasing the IO performances, this commit introduces a new function
queues_per_thread() to the VhostUserBackend trait.
This gives each backend implementation the opportunity to define which
queues belong to which thread.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By changing the mutability of this function, after adapting all
backends, we should be able to implement multithreads with
multiqueues support without hitting a bottleneck on the backend
locking.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Both blk, net and fs backends have been updated to avoid the requirement
of having handle_event(&mut self). This will allow the backend crate to
avoid taking a write lock onto the backend object, which will remove the
potential contention point when multiple threads will be handling
multiqueues.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
While testing self spawned vhost-user backends, it appeared that the
backend was aborting due to a missing system call in the seccomp
filters. mremap() was the culprit and this patch simply adds it to the
whitelist.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This should make it a pointer in the Go generated code so that it will
be ommitted and thus not populated with an unhelpful default value.
Fixes: #1015
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This ensures that the field is filled with None when it is not specified
as part of the deserialisation step.
Fixes: #1015
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
We can now allow guests that specify an initramfs to boot
using the PVH boot protocol.
Signed-off-by: Alejandro Jimenez <alejandro.j.jimenez@oracle.com>
Fill and write to guest memory the necessary boot module
structure to allow a guest using the PVH boot protocol
to load an initramfs image.
Signed-off-by: Alejandro Jimenez <alejandro.j.jimenez@oracle.com>
This allows the validation of this requirement for both command line
booted VMs and those booted via the API.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This allows the validation of this requirement for both command line
booted VMs and those booted via the API.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
As the VmConfig::Parse() also does validation work it only make sense to
parse the VM options on the VM boot path only.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Replace the existing VmConfig::valid() check with a call into
.validate() as part of earlier config setup or boot API checks.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When performing an API boot validate the configuration. For now only
some very basic validation is performed but in subsequent commits
the validation will be extended.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The configuration comes from a variety of places (commandline, REST API
and restore) however some validation was only happening on the command
line parsing path. Therefore introduce a new ability to validate the
configuration before proceeding so that this can be used for commandline
and API boots.
For now move just the console and serial output mode validation under
the new validation API.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Making sure the OpenAPI definition is up to date with newly added
structure and parameters to support VM restoration.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that the restore path uses RestoreConfig structure, we add a new
parameter called "prefault" to it. This will give the user the ability
to populate the pages corresponding to the mapped regions backed by the
snapshotted memory files.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The goal here is to move the restore parameters into a dedicated
structure that can be reused from the entire codebase, making the
addition or removal of a parameter easier.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
When CoW can be used, the VM restoration time is reduced, but the pages
are not populated. This can lead to some slowness from the guest when
accessing these pages.
Depending on the use case, we might prefer a slower boot time for better
performances from guest runtime. The way to achieve this is to prefault
the pages in this case, using the MAP_POPULATE flag along with CoW.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This patch extends the previous behavior on the restore codepath.
Instead of copying the memory regions content from the snapshot files
into the new memory regions, the VMM will use the snapshot region files
as the backing files behind each mapped region. This is done in order to
reduce the time for the VM to be restored.
When the source VM has been initially started with a backing file, this
means it has been mapped with the MAP_SHARED flag. For this case, we
cannot use the CoW trick to speed up the VM restore path and we simply
fallback onto the copy of the memory regions content.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Whenever a MemoryManager is restored from a snapshot, the memory regions
associated with it might need to directly back the mapped memory for
increased performances. If that's the case, a list of external regions
is provided and the MemoryManager should simply ignore what's coming
from the MemoryConfig.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that we can choose specific mmap flags for the guest RAM, we create
a new parameter "copy_on_write" meaning that the memory mappings backed
by a file should be performed with MAP_PRIVATE instead of MAP_SHARED.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to anticipate the need for special mmap flags when memory
mapping the guest RAM, we need to switch from from_file() wrapper to
build() wrapper.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
A few KVM ioctls were missing in order to perform both snapshot and
restore while keeping seccomp enabled.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This connects the dots together, making the request from the user reach
the actual implementation for restoring the VM.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This connects the dots together, making the request from the user reach
the actual implementation for snapshotting the VM.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The MemoryManager is somehow a special case, as its restore() function
was not implemented as part of the Snapshottable trait. Instead, and
because restoring memory regions rely both on vm.json and every memory
region snapshot file, the memory manager is restored at creation time.
This makes the restore path slightly different from CpuManager, Vcpu,
DeviceManager and Vm, but achieve the correct restoration of the
MemoryManager along with its memory regions filled with the correct
content.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
This is only implementing the send() function in order to store all Vm
states into a file.
This needs to be extended for live migration, by adding more transport
methods, and also the recv() function must be implemented.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
By aggregating snapshots from the CpuManager, the MemoryManager and the
DeviceManager, Vm implements the snapshot() function from the
Snapshottable trait.
And by restoring snapshots from the CpuManager, the MemoryManager and
the DeviceManager, Vm implements the restore() function from the
Snapshottable trait.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
This implements the send() function of the Transportable trait, so that
the guest memory regions can be saved into one file per region.
This will need to be extended for live migration, as it will require
other transport methods and the recv() function will need to be
implemented too.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
In order to snapshot the content of the guest RAM, the MemoryManager
must implement the Snapshottable trait.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Implement the Snapshottable trait for Vcpu, and then implements it for
CpuManager. Note that CpuManager goes through the Snapshottable
implementation of Vcpu for every vCPU in order to implement the
Snapshottable trait for itself.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Anticipating the need for a slightly different function for restoring
vCPUs, this patch factorizes most of the vCPU creation, so that it can
be reused for migration purposes.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
These two new helpers will be useful to capture a vCPU state and being
able to restore it at a later time.
Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
In anticipation for the CpuManager to aggregate all Vcpu snapshots
together, this change makes sure the CpuManager has a handle onto
every vCPU.
Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Based on the list of Migratable devices stored by the DeviceManager, the
DeviceManager can implement the Snapshottable trait by aggregating all
devices snapshots together.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Serial and Ioapic both implement the Migratable trait, hence the
DeviceManager can store them in the list of Migratable devices.
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
We need the project to rely on kvm-bindings and kvm-ioctls branches
which include the serde derive to be able to serialize and deserialize
some KVM structures.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The parse_size helper function can now be consolidated into the
ByteSized FromStr implementation.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Now all parsing code makes use of the Toggle and it's FromStr support
move the helper function into the from_str() implementation.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The integration tests and documentation make use of empty value strings
like "--net tap=" accept them but return None so that the default value
will be used as expected.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This simplifies the parsing of the option by using OptionParser along
with its automatic conversion behaviour.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Byte sizes are quantities ending in "K", "M", "G" and by implementing
this type with a FromStr implementation the values can be converted
using .parse().
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Before porting over to OptionParser add a unit test to validate the
current memory parsing code. This showed up a bug where the "size=" was
always required. Temporarily resolve this by assigning the string a
default value which will later be replaced when the code is refactored.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This will be used to simplify and consolidate much of the parsing code
used for command line parameters.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
A Snapshottable component can snapshot itself and
provide a MigrationSnapshot payload as a result.
A MigrationSnapshot payload is a map of component IDs to a list of
migration sections (MigrationSection). As component can be made of
several Migratable sub-components (e.g. the DeviceManager and its
device objects), a migration snapshot can be made of multiple snapshot
itself.
A snapshot is a list of migration sections, each section being a
component state snapshot. Having multiple sections allows for easier and
backward compatible migration payload extensions.
Once created, a migratable component snapshot may be transported and this
is what the Transportable trait defines, through 2 methods: send and recv.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
On some systems, the open() system call is used by Cloud-Hypervisor,
that's why it should be part of the seccomp filters whitelist.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Both clock_gettime and gettimeofday syscalls where missing when running
Cloud-Hypervisor on a Linux host without vDSO enabled. On a system with
vDSO enabled, the syscalls performed by vDSO were not filtered, that's
why we didn't have to whitelist them.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Return libc::EINVAL instead of custom "Wrong offset" error, as mmap(2)
returns EINVAL when offset/len is invalid.
Signed-off-by: Eryu Guan <eguan@linux.alibaba.com>
In fs_slave_map/unmap/sync, we only made sure offset < cache_size, but
didn't validate (offset + len). We should ensure [offset, offset+len]
is within cache range as well.
Signed-off-by: Eryu Guan <eguan@linux.alibaba.com>
One more time, we're extending the VFIO integration test to verify that
a VFIO device passed through a VM is still usable after some new memory
has been hotplugged.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Extend the update_memory() method from DeviceManager so that VFIO PCI
devices can update their DMA mappings to the physical IOMMU, after a
memory hotplug has been performed.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Whenever the memory is resized, it's important to retrieve the new
region to pass it down to the device manager, this way it can decide
what to do with it.
Also, there's no need to use a boolean as we can instead use an Option
to carry the information about the region. In case of virtio-mem, there
will be no region since the whole memory has been reserved up front by
the VMM at boot. This means only the ACPI hotplug will return a region
and is the only method that requires the memory to be updated from the
device manager.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Whenever the guest memory is extended, the IOMMU mappings related to
each VFIO device must be updated. This allows subsequent RAM accesses
to be authorized by the physical IOMMU for the newly added RAM region.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Commit 2adddce2 reorganized the crate for a cleaner multi architecture
(x86_64 and aarch64) support.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
For now, the codebase does not support booting from initramfs with PVH
boot protocol, therefore we need to fallback to the legacy boot.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
the integration test creates an initramfs image based on AlpineLinux mini root filesystem
with a simple /init script that just echoes a string to the console. The string
is passed via the kernel cmdline as an environment variable.
Signed-off-by: Damjan Georgievski <gdamjan@gmail.com>
* load the initramfs File into the guest memory, aligned to page size
* finally setup the initramfs address and its size into the boot params
(in configure_64bit_boot)
Signed-off-by: Damjan Georgievski <gdamjan@gmail.com>
currently unused, the initramfs argument is added to the cli,
and stored in vmm::config:VmConfig as an Option(InitramfsConfig(PathBuf))
Signed-off-by: Damjan Georgievski <gdamjan@gmail.com>
The persistent memory will be hotplugged via DeviceManager and saved in
the config for later use.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Split it into a method that creates a single device which is called by
the multiple device version so this can be used when dynamically adding
a device.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This commit adds new option hotplug_method to memory config.
It can set the hotplug method to "acpi" or "virtio-mem".
Signed-off-by: Hui Zhu <teawater@antfin.com>
The basic idea of virtio-mem is to provide a flexible, cross-architecture
memory hot plug and hot unplug solution that avoids many limitations
imposed by existing technologies, architectures, and interfaces. More
details can be found in https://lkml.org/lkml/2019/12/12/681.
This commit add virtio-mem device.
Signed-off-by: Hui Zhu <teawater@antfin.com>
The persistent memory will be hotplugged via DeviceManager and saved in
the config for later use.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Split it into a method that creates a single device which is called by
the multiple device version so this can be used when dynamically adding
a device.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
We made sure gpa is in cache range, but not the end addr of request,
which is (gpa + len). If the end addr of request is beyond dax cache
window, vmm would corrupt guest memory or crash.
Fix it by making sure end addr of request is within cache range as well.
And while we're at it, return EFAULT if the request is out of range, as
write(2)/read(2) returns EFAULT when buffer is outside accessible
address space.
Signed-off-by: Eryu Guan <eguan@linux.alibaba.com>
Split it into a method that creates a single device which is called by
the multiple device version so this can be used when dynamically adding
a device.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
All vhost-user-blk, vhost-user-net and virtio-fs integration tests are
extended with memory resizing. The point is to validate that we can
hotplug some memory and keep these backends to work as expected.
Just a note that because memory resizing is not supported by mmio, we
limit these extensions to all non-mmio use cases.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The same Clear Linux kernel command line was repeated again and again in
the code. That's why this commit takes care of factorizing it, which
simplifies the code but also simplify maintenance whenever we'd need to
update the block UUID.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The hypervisor-fw does not support virtio-blk through mmio transport
layer, therefore we can only run tests with mmio if these tests boot
directly to kernel.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
There's no reason mmio builds cannot support vhost-user-blk, therefore
there's no reason to skip these tests for mmio.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
As a first step towards factorizing as many vhost-user-blk tests as
possible, this patch takes care of unifying their expected output.
It also makes a clear separation between the tests booting from
vhost-user-blk from the ones passing an extra volume.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to limit code duplication, vhost-user-net integration tests are
further factorized, as they now include the self spawning test.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to keep vhost-user backend to work across guest memory resizing
happening when memory is hot-plugged or hot-unplugged, both blk, net and
fs devices are implementing the notifier to let the backend know.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By factorizing the setup of the memory table for vhost-user, we
anticipate the fact that vhost-user devices are going to reuse this
function when the guest memory will be updated.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Whenever the VM memory is resized, DeviceManager needs to be notified
so that it can subsequently notify each virtio devices about it.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The virtio devices backed by a vhost-user backend must send an update to
the associated backend with the new file descriptors corresponding to
the memory regions.
This patch allows such devices to be notified when such update needs to
happen.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This separates the filters used between the VMM and API threads, so that
we can apply different rules for each thread.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This change introduces a new CLI option --seccomp. This allows the user
to enable/disable the seccomp filters when needed. Because the user now
has the possibility to disable the seccomp filters, and because the
Cloud-Hypervisor project wants to enforce the maximum security by
default, the seccomp filters are now applied by default.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit introduces the application of the seccomp filter to the VMM
thread. The filter is empty for now (SeccompLevel::None).
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the seccomp crate, we create a new vmm module responsible for
creating a seccomp filter that will be applied to the VMM main thread.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The seccomp crate from Firecracker is nicely implemented, documented and
tested, which is a good reason for relying on it to create and apply
seccomp filters.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Virtio-fs daemon expects fs_slave_io() returns the number of bytes
read/written on success, but we always return 0 and make userspace think
nothing has been read/written.
Fix it by returning the actual bytes read/written. Note that This
depends on the corresponding fix in vhost crate.
Fixes: #949
Signed-off-by: Eryu Guan <eguan@linux.alibaba.com>
Factorise test_virtio_pmem to test changes are not persisted if file is
readonly.
As virtio-mem is now supported in the upstream kernel we can switch to
using the firmware.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Rather than using a raw OS disk image. This will be useful when the test
is extended to doing I/O on the image.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This opens the backing file read-only, makes the pages in the mmap()
read-only and also makes the KVM mapping read-only. The file is also
mapped with MAP_PRIVATE to make the changes local to this process only.
This is functional alternative to having support for making a
virtio-pmem device readonly. Unfortunately there is no concept of
readonly virtio-pmem (or any type of NVDIMM/PMEM) in the Linux kernel so
to be able to have a block device that is appears readonly in the guest
requires significant specification and kernel changes.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Use this boolean to turn on the KVM_MEM_READONLY flag to indicate that
this memory mapping should not be writable by the VM.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
According to `asm-generic/termios.h`, the `struct winsize` should be:
struct winsize {
unsigned short ws_row;
unsigned short ws_col;
unsigned short ws_xpixel;
unsigned short ws_ypixel;
};
The ioctl of TIOCGWINSZ will trigger a segfault on aarch64.
Signed-off-by: Qiu Wenbo <qiuwenbo@phytium.com.cn>
This feature is stable and there is no need for this to be behind a
flag. This will also reduce the time needed to run the integration test
as we will not be running them all again under the flag.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This table currently contains only all the VFIO devices and it should
really contain all the PCI devices.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Previously this was only returned if the device had an IOMMU mapping and
whether the device should be added to the virtio-iommu. This was already
captured earlier as part of creating the device so use that information
instead.
Always returning the B/D/F is helpful as it facilitates virtio PCI
device hotplug.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Implement missing support for FUSE_LSEEK, which basically implies
calling to libc::lseek on the file handle. As this operation alters
the file offset, we take a write lock on the File's RwLock.
Signed-off-by: Sergio Lopez <slp@redhat.com>
Add a document to illustrate how to test vhost-user-net function with
OpenVSwitch + DPDK as backend.
Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
On x86_64, a hint to the compiler is not enough, we need to issue a
MFENCE instruction. Replace the Acquire fence with a SeqCst one.
Without this, it's still possible to miss an used_event update,
leading to the omission of a notification, possibly stalling the
vring.
Signed-off-by: Sergio Lopez <slp@redhat.com>
I spent a few minutes trying to understand why we were unconditionally
updating the VM config memory size, even if the guest memory resizing
did not happen.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
We set it to 0xff, which is for unregistered loaders.
The kernel checks that the bootloader ID is set when e.g. loading
ramdisks, so not setting it when we get a bootparams header from the
loader will prevent the kernel from loading ramdisks.
Fixes: #918
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The IORT table for virtio-iommu use was removed and replaced with a
purely virtio based solution. Although the table construction was
removed these structures were left behind.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Virtiofs's dax window can be used as read/write's source (e.g. mmap a file
on virtiofs), but the dax window area is not shared with vhost-user
backend, i.e. virtiofs daemon.
To make those IO work, addresses of this kind of IO source are routed to
VMM via FS_IO requests to perform a read/write from an fd directly to the
given GPA.
This adds the support of FS_IO request to clh's vhost-user-fs master part.
Signed-off-by: Liu Bo <bo.liu@linux.alibaba.com>
Add an integration test that builds cloud-hypervisor with
the pvh_boot feature and boots a kernel built with CONFIG_PVH.
Signed-off-by: Alejandro Jimenez <alejandro.j.jimenez@oracle.com>
Validate correct GDT entries, initial segment configuration, and control
register bits that are required by PVH boot protocol.
Signed-off-by: Alejandro Jimenez <alejandro.j.jimenez@oracle.com>
Expand the unit tests to cover the configure_system() code when
using the PVH boot protocol. Verify the method for adding memory
map table entries in the format specified by PVH boot protocol.
Signed-off-by: Alejandro Jimenez <alejandro.j.jimenez@oracle.com>
Use a new feature called "pvh_boot" to enable using the PVH boot
protocol if the guest kernel supports it. The feature can be enabled
by building with:
cargo build [--release] --features "pvh_boot"
Once performance has been evaluated, this can be made part of the
default set of features so that any guest that supports it boots
using PVH as the preferred option as is the case in QEMU.
Signed-off-by: Alejandro Jimenez <alejandro.j.jimenez@oracle.com>
Fill the hvm_start_info and related memory map structures as
specified in the PVH boot protocol. Write the data structures
to guest memory at the GPA that will be stored in %rbx when
the guest starts.
Signed-off-by: Alejandro Jimenez <alejandro.j.jimenez@oracle.com>
In order to properly initialize the kvm regs/sregs structs for
the guest, the load_kernel() return type must specify which
boot protocol to use with the entry point address it returns.
Make load_kernel() return an EntryPoint struct containing the
required information. This structure will later be used
in the vCPU configuration methods to setup the appropriate
initial conditions for the guest.
Signed-off-by: Alejandro Jimenez <alejandro.j.jimenez@oracle.com>
Create supporting definitions to use the hvm start info and memory
map table entry struct definitions from the linux-loader crate in
order to enable PVH boot protocol support
Signed-off-by: Alejandro Jimenez <alejandro.j.jimenez@oracle.com>
@dependabot bumped the dependency to 0.4.10 but this is no longer a
valid version so downgrade appropriately.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Extended attributes (xattr) support has a huge impact on write
performance. The reason for this is that, if enabled, FUSE sends a
setxattr request after each write operation, and due to the inode
locking inside the kernel during said request, the ability to execute
the operations in parallel becomes heavily limited.
Signed-off-by: Sergio Lopez <slp@redhat.com>
This change enables vhost_user_fs to process multiple requests in
parallel by scheduling them into a ThreadPool (from the Futures
crate).
Parallelism on a single file is limited by the nature of the operation
executed on it. A recent commit replaced the Mutex that protects the
File within HandleData with a RwLock, to allow some operations (at
this moment, only "read" and "write") to proceed in parallel by
acquiring a read lock.
A more complex approach was also implemented [1], involving
instrumentation through vhost_user_backend to be able to serialize
completions, reducing the pressure on the vring RwLock. This strategy
improved the performance on some corner cases, while making it worse
on other, more common ones. This fact, in addition to it requiring
wider changes through the source code, prompted me to drop it in favor
of this one.
[1] https://github.com/slp/cloud-hypervisor/tree/vuf_async
Signed-off-by: Sergio Lopez <slp@redhat.com>
"DescriptorChain"s are tied to the lifetime of the referenced
GuestMemoryMmap object (for good reasons), but sometimes (i.e., when
processing descriptors from different contexts) we may need to switch
them to point a different GuestMemoryMmap.
Here we introduce the structure DescriptorHead, which holds the data
needed to rebuild a DescriptorChain, the method "get_head" which
returns the DescriptorHead for a DescriptorChain, and the method
"new_from_head", which allows to create a new DescriptorChain with a
DescriptorHead and a new reference to a GuestMemoryMmap.
Signed-off-by: Sergio Lopez <slp@redhat.com>
Replace HandleData's File Mutex with a RwLock to have more granularity
on the lock. This allows operations on the same File that are safe to
be run in parallel (at this moment, read and write), to acquire a read
lock to avoid waiting on each other.
Signed-off-by: Sergio Lopez <slp@redhat.com>
When using "--disk" with a vhost socket and not using self spawning then
it is not necessary or helpful to specify the path.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This option was superseded by using "--net" with "vhost_user=true". This
option wasn't being parsed any more but was left over.
Fixes: #806
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
By using a Vec to hold the list of devices on the PciBus, there's a
problem when we use unplug. Indeed, the vector of devices gets reduced
and if the unplugged device was not the last one from the list, every
other device after this one is shifted on the bus.
To solve this problem, a HashMap is used. This allows to keep track of
the exact place where each device stands on the bus.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The option desired_ram is in byte, make larger the amount of memory to
add.
Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
Some of the help strings had extra newlines in them or otherwise strange
wrapping. The strings were rewrapped with the nightly version of rustfmt
that supports string formatting.
Fixes: #899
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This prevents the output being wrapped at 120 characters and giving
strange results.
Fixes: #899
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This change, combined with the compiler hint to inline get_used_event,
shortens the window between the memory read and the actual check by
calling get_used_event from needs_notification.
Without it, when putting enough pressure on the vring, it's possible
that a notification is wrongly omitted, causing the queue to stall.
Signed-off-by: Sergio Lopez <slp@redhat.com>
get_used_event is used from vhost_user_backend:needs_notification to
check whether an interrupt must be sent to the guest to notify there
are new items in the queue. Shorten the update window by asking the
the compiler to inline this method, so a write won't slip between the
read of the memory contents and the actual check.
Signed-off-by: Sergio Lopez <slp@redhat.com>
Since we only keep one single version of the kernel config file in our
repository, there is no reason to keep the filename complex.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The kernel version is updated from 5.5-rc1 to 5.6-rc4, including the
updated kernel config file.
The kernel branch contains virtio-fs, virtio-iommu and virtio-mem
patches that are not upstream yet. It also contains one fix for
virtio-vsock which will be merged upstream in the next release.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We define 1 workflow with 2 jobs: One for the regular build and another
one for creating a release and uploading the corresponding assets.
They both run conditionally, depending on the gihtub event that triggered
the action.
Fixes: #825
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
With some of the factorization that happened to be able to support VFIO
hotplug, one mistake was made. In case a vIOMMU is created through a
virtio-iommu device, and no matter the "iommu" option value from the
VFIO device parameter, the VFIO device was always placed behind the
virtual IOMMU.
This commit fixes this wrong behavior by making sure the device
configuration is taken into account to decide if it should be attached
or not to the virtual IOMMU.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Add a new id option to the VFIO hotplug command so that it matches the
VFIO coldplug semantic.
This is done by refactoring the existing code for VFIO hotplug, where
VmAddDeviceData structure is replaced by DeviceConfig. This structure is
the one used whenever a VFIO device is coldplugged, which is why it
makes sense to reuse it for the hotplug codepath.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Add the ability to specify the "id" associated with a device, by adding
an extra option to the parameter --device.
This new option is not mandatory, and by default, the VMM will take care
of finding a unique identifier.
If the identifier provided by the user through this new option is not
unique, an error will be thrown and the VM won't be started.
Fixes#881
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The 32 bits MMIO address space is handled separately from the 64 bits
one. For this reason, we need to invoke the appropriate freeing function
to remove a range from this address space.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that PciDevice trait has a dedicated function to remove the bars,
the DeviceManager can invoke this function whenever a PCI device is
unplugged from the VM.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Whenever a VfioPciDevice is dropped, the regions previously mapped are
being unmapped. But there's also a need for removing the region from a
KVM perspective.
This commit extends the existing unmap_mmio_regions() function to take
care of removing the KVM region at the same time the regions are being
unmapped.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to provide the tools for a complete cleanup whenever a VFIO PCI
device is removed from the VM, the VfioPciDevice implements free_bars()
method from PciDevice trait. This will take care of removing the IO and
MMIO ranges previously reserved through the vm-allocator.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The allocator already had functions to allocate and free both IO and 64
bits MMIO address spaces, but it only had an allocating function for 32
bits MMIO address space.
With this new function, it will be now possible to remove cleanly some
ranges from the 32 bits MMIO address space.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The point of this new method is to let the caller decide when the
implementation of the PciDevice should free the BARs previously
allocated through the other method allocate_bars().
This provides a way to perform proper cleanup for any PCI device.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Support sending a request body this will usually be JSON encoded data
representing the details of the request.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Upon removal of a PCI device, make sure we don't hold onto the device ID
as it could be reused for another device later.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to handle the case where devices are very often plugged and
unplugged from a VM, we need to handle the PCI device ID allocation
better.
Any PCI device could be removed, which means we cannot simply rely on
the vector size to give the next available PCI device ID.
That's why this patch stores in memory the information about the 32
slots availability. Based on this information, whenever a new slot is
needed, the code can correctly provide an available ID, or simply return
an error because all slots are taken.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit ensures that when a VFIO device is hot-unplugged from the
VM, it is also removed from the VmConfig. This prevents a potential
reboot from creating the device.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Add a new field to the DeviceConfig, allowing the VMM to allocate a name
to the VFIO devices.
By identifying a VFIO device with a unique name, we can make sure a user
can properly unplug it at any time.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit introduces the new command "remove-device" that will let a
user hot-unplug a VFIO PCI device from an already running VM.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit implements the eject function so that a VFIO device will be
removed from any bus it might sit on, and from any list it might be
stored in.
The idea is to reach a point where there is no reference of the device
anywhere in the code, so that the Drop implementation will be invoked
and so that the device will be fully removed from the VMM.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
When the guest OS is done removing a PCI device, it will invoke the _EJ0
method from ACPI, associated with the device. This will trigger a port
IO write to a region known by the VMM. Upon this writing, the VMM will
trap the VM exit and retrieve the written value.
Based on the value, the VMM will invoke its eject_device() method to
finalize the removal of the device.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The same BusDevice can be inserted multiple times on the same bus at
different ranges. This is the case for a PCI device with multiple memory
BARs for example, as each BAR will be inserted at a specific MMIO range.
This new method allows to find and remove every occurence of the same
device so that we can make sure it is fully removed from the bus.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Simple function relying on the retain() method from std::Vec, allowing
to remove every occurence of the same device.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
As we try to keep track of every PCI device related to the VM, we don't
want to have separate lists depending on the concrete type associated
with the PciDevice trait. Also, we want to be able to cast the actual
type into any trait or concrete type.
The most efficient way to solve all these issues is to store every
device as an Arc<dyn Any + Send + Sync>. This gives the ability to
downcast into the appropriate concrete type, and then to cast back into
any trait that we might need.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Add a new list storing the device names across the entire codebase. VFIO
devices are added to the list whenever a new one is created. By default,
each VFIO device is given a name "vfioX" where X is the first available
integer.
Along with this new list of names, another list is created, grouping PCI
device's name with its associated b/d/f. This will be useful to keep
track of the created devices so that we can implement unplug
functionality.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
When a virtio device is paused an event is written to the appropriate
"pause" EventFd for the device. This will be noticed by the the device's
epoll_wait(), an atomic bool checked an if true then the thread is
parked(). When resuming the bool is reset and the thread is unpark()ed.
However the event triggering the pause is still in the EventFd so the
epoll_wait() will continue to return but because the boolean is not set
the thread will not be park()ed but instead we will busy loop around an
event that is not being consumed.
The solution is to drain the "pause" EventFd when the event is first
received and thus the epoll_wait() will only return for the pause event
once. This resolves the infinite epoll_wait() wake-ups.
Fixes: #869
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This commit introduces a basic implementation of a remote control of a
running VMM implementing a subset of the API.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
If a new build request for master comes in don't cancel the older
builds. Cancelling older builds is very useful for PRs that are being
updated but less so for a build of the master branch that has been
triggered via a merge or a periodic build.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
For cases where IOMMU is not supported, iommu_group will be no present.
This could happened because IOMMU is off at kernel level, bios level or
is not supported at all.
Instead of fail with a generic error handle it as different use case:
Before:
ReadLink(Os { code: 2, kind: NotFound, message: "No such file or directory"})
After:
VfioNoIommuGroup("/sys/bus/pci/devices/0000:03:00.0/iommu_group")
Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
Replace sudo invocations of bash creating a subshell with simpler
solutions by running the program directly on the input or using tee.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
As part of bringing the tests up with virtio-mmio the virtio-fs were
wrongly disabled. Re-enable them and remove the cache size check as the
cache does not appear in /proc/iomem on virtio-mmio.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When booting from vhost-user-block the entropy is sometimes lower
triggering a flaky test. We have already use other, more reliable
methods for checking that the VM has booted.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The Vm structure was used to store a strong reference to the IO bus.
This is not needed anymore since the AddressManager is logically the
one holding this strong reference. This has been made possible by the
introduction of Weak references on the Bus structure itself.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that the BusDevice devices are stored as Weak references by the
IO and MMIO buses, there's no need to use Weak references from the
DeviceManager anymore.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that the BusDevice devices are stored as Weak references by the
IO and MMIO buses, there's no need to use Weak references from the
CpuManager anymore.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that the BusDevice devices are stored as Weak references by the IO
and MMIO buses, there's no need to use Weak references from the PciBus
anymore.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By storing a list of Weak references to BusDevice devices, the Bus
structure prevents potential cyclic dependencies from happening on
strong references.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The point is to make sure the DeviceManager holds a strong reference of
each BusDevice inserted on the IO and MMIO buses. This will allow these
buses to hold Weak references onto the BusDevice devices.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The method add_vfio_device() from the DeviceManager needs to be mutable
if we want later to be able to update some internal fields from the
DeviceManager from this same function.
This commit simply takes care of making the necessary changes to change
this function as mutable.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
It's more logical to name the field referring to the DeviceManager as
"device_manager" instead of "devices".
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit extends the existing test_vfio by hotplugging an extra
virtio-net device to the L2 VM. The test for validating the hotplug
succeeded is the same as the one to verify the non-hotplugged devices.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By inserting the DeviceManager on the IO bus, we introduced some cyclic
dependency:
DeviceManager ---> AddressManager ---> Bus ---> BusDevice
^ |
| |
+---------------------------------------------+
This cycle needs to be broken by inserting a Weak reference instead of
an Arc (considered as a strong reference).
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Ensures the configuration is updated after a new device has been
hotplugged. In the event of a reboot, this means the new VM will be
started with the new device that had been previously hotplugged.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit finalizes the VFIO PCI hotplug support, based on all the
previous commits preparing for it.
One thing to notice, this does not support vIOMMU yet. This means we can
hotplug VFIO PCI devices, but we cannot attach them to an existing or a
new virtio-iommu device.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This factorization is very important as it will allow both the standard
codepath and the VFIO PCI hotplug codepath to rely on the same function
to perform the addition of a new VFIO PCI device.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Whenever the user wants to hotplug a new VFIO PCI device, the VMM will
have to trigger a hotplug notification through the GED device.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit introduces the new command "add-device" that will let a user
hotplug a VFIO PCI device to an already running VM.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Through the BusDevice implementation from the DeviceManager, and by
inserting the DeviceManager on the IO bus for a specific IO port range,
the VMM now has the ability to handle PCI device hotplug.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In anticipation of inserting the DeviceManager on the IO/MMIO buses,
the DeviceManager must implement the BusDevice trait.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Expands the existing GED device description in the DSDT table. A new
case of hotplug notification is being added to the already existing CPU
and memory hotplug. The third case of hotplug being the addition/removal
of a PCI device, it is identified through the flag 0x4.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Create a small method that will perform both hotplug of all the devices
identified by PCIU bitmap, and then perform the hotunplug of all the
devices identified by the PCID bitmap.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The _EJ0 method provides the guest OS a way to notify the VMM that the
device has been properly ejected from the guest OS. Only after this
point, the VMM can fully remove the device.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This new PHPR device in the DSDT table introduces some specific
operation regions and the associated fields.
PCIU stands for "PCI up", which identifies PCI devices that must be
added.
PCID stands for "PCI down", which identifies PCI devices that must be
removed.
B0EJ stands for "Bus 0 eject", which identifies which device on the bus
has been ejected by the guest OS.
Thanks to these fields, the VMM and the guest OS can communicate while
performing hotplug/hotunplug operations.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Adds the DVNT method to the PCI0 device in the DSDT table. This new
method is responsible for checking each slot and notify the guest OS if
one of the slots is supposed to be added or removed.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit introduces the ACPI support for describing the 32 device
slots attached to the main PCI host bridge.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that virtio-mmio transport has support for SHM regions we should be
able to test virtio-fs against it.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The details of the SHM regions or the lack of, which is used by
virtio-fs DAX, is communicated through configuration fields on the
virtio-mmio memory region. Implement the necessary fields to return the
SHM entries and in particular return a length of (u64)-1 which is used
by the kernel to indicate there are no SHM regions.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
A new ClearLinux image has been uploaded to the Azure storage account.
It is based off of the ClearLinux cloudguest image 31310 version, with
three extra bundles added to it.
First bundle is curl, which adds the curl binary to the image, second
bundle is iperf, adding the iperf binary to the image, and third bundle
is sysadmin-basic to include utility like netcat.
The image is 2G in size.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
As we switched to external vhost_rs crate, delete this internal one.
All vhost changes should go to cloud-hypervisor/vhost.
Signed-off-by: Eryu Guan <eguan@linux.alibaba.com>
As cloud-hypervisor/vhost crate (dragonball branch) is ready to be used,
switch vhost_rs from internal crate to the external one.
Signed-off-by: Eryu Guan <eguan@linux.alibaba.com>
Current device configuration space offset value is 0, we need to
update that value to VHOST_USER_CONFIG_OFFSET(0x100) to follow the spec
Fixes#844
Signed-off-by: Arron Wang <arron.wang@intel.com>
On tests that expect a clean shutdown there is no need to try and kill
the child after wait() has returned as the process has already exited.
Further there is no need to sleep before wait() as wait will block until
the VM and VMM shutdown is complete.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This should address any flakiness as the VMM process will have
completely terminated and all files closed.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Many of the tests attempted to SSH into the VM and then run "shutdown"
but don't actually check that the VM has shutdown correctly and proceed
to kill the child process. Remove the associated SSH commands and sleeps
from those tests that are not explicitly checking the shutdown
behaviour.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Only some tests require the output for the tests to be captured so
default to not capturing the output to a pipe and instead make it
controllable.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Many of the tests use an identical network configuration. Add a
GuestCommand::default_net() to generate this configuration and use it
wherever possible.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Many of the tests use an identical disk configuration. Add a
GuestCommand::default_disks() to generate this configuration and use it
wherever possible.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This is a thin wrapper over std::process:Command which currently only
specifies the default binary but in future will handle more default
behaviour.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In anticipation of the support for device hotplug, this commit moves the
DeviceManager object into an Arc<Mutex<>> when the DeviceManager is
being created. The reason is, we need the DeviceManager to implement the
BusDevice trait and then provide it to the IO bus, so that IO accesses
related to device hotplug can be handled correctly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We weren't processing events arriving at the HIPRIO queue, which
implied ignoring FUSE_INTERRUPT, FUSE_FORGET, and FUSE_BATCH_FORGET
requests.
One effect of this issue was that file descriptors weren't closed on
the server, so it eventually hits RLIMIT_NOFILE. Additionally, the
guest OS may hang while attempting to unmount the filesystem.
Signed-off-by: Sergio Lopez <slp@redhat.com>
There is no reason to give some special capabilities to the Rust version
of virtiofsd since it behaves slightly differently and does not require
neither DAC_OVERRIDE nor SYS_ADMIN.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
vhost_user_fs doesn't really support all vhost protocol features, just
MQ and SLAVE_REQ, so return that in protocol_features().
Signed-off-by: Sergio Lopez <slp@redhat.com>
Indirect descriptors is a virtio feature that allows the driver to
store a table of descriptors anywhere in memory, pointing to it from a
virtqueue ring's descriptor with a particular flag.
We can't seamlessly transition from an iterator over a conventional
descriptor chain to an indirect chain, so Queue users need to
explicitly support this feature by calling Queue::is_indirect() and
Queue::new_from_indirect().
Signed-off-by: Sergio Lopez <slp@redhat.com>
We import slave_fs_cache mod under vhost-user-slave feature control,
but not the self::slave_fs_cache::SlaveFsCacheReq import.
Signed-off-by: Eryu Guan <eguan@linux.alibaba.com>
We want to prevent from losing interrupts while they are masked. The
way they can be lost is due to the internals of how they are connected
through KVM. An eventfd is registered to a specific GSI, and then a
route is associated with this same GSI.
The current code adds/removes a route whenever a mask/unmask action
happens. Problem with this approach, KVM will consume the eventfd but
it won't be able to find an associated route and eventually it won't
be able to deliver the interrupt.
That's why this patch introduces a different way of masking/unmasking
the interrupts, simply by registering/unregistering the eventfd with the
GSI. This way, when the vector is masked, the eventfd is going to be
written but nothing will happen because KVM won't consume the event.
Whenever the unmask happens, the eventfd will be registered with a
specific GSI, and if there's some pending events, KVM will trigger them,
based on the route associated with the GSI.
Suggested-by: Liu Jiang <gerry@linux.alibaba.com>
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We should not assume the offset produced by ECAM is identical to the
CONFIG_ADDRESS register of legacy PCI port io enumeration.
Signed-off-by: Qiu Wenbo <qiuwenbo@phytium.com.cn>
This option improves the security of the guest by randomising the start
address of the kernel in physical memory. We should turn this on so as
to ensure all our functionality such as memory hotplug and kernel
loading works as this is an option used widely in production.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Recently, vhost_user_block gained the ability of actively polling the
queue, a feature that can be disabled with the poll_queue property.
This change adds this property to DiskConfig, so it can be used
through the "disk" argument.
For the moment, it can only be used when vhost_user=true, but this
will change once virtio-block gets the poll_queue feature too.
Fixes: #787
Signed-off-by: Sergio Lopez <slp@redhat.com>
Fix "readonly" and "wce" defaults in cloud-hypervisor.yaml to match
their respective defaults in config.rs:DiskConfig.
Signed-off-by: Sergio Lopez <slp@redhat.com>
This is a perfectly acceptable situation as it causes the backend to
exit because the VMM has closed the connection. This addresses the
rather ugly reporting of errors from the backend that appears
interleaved with the output from the VMM.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Return an error wen recvmsg() returns without a message using the
libc::ECONNRESET error so that the upper levels will correctly
interpret this as the connection being broken.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
It's missing a few knobs (readonly, vhost, wce) that should be exposed
through the rest API.
Fixes: #790
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The kernel does not adhere to the ACPI specification (probably to work
around broken hardware) and rather than busy looping after requesting an
ACPI reset it will attempt to reset by other mechanisms (such as i8042
reset.)
In order to trigger a reset the devices write to an EventFd (called
reset_evt.) This is used by the VMM to identify if a reset is requested
and make the VM reboot. As the reset_evt is part of the VMM and reused
for both the old and new VM it is possible for the newly booted VM to
immediately get reset as there is an old event sitting in the EventFd.
The simplest solution is to "drain" the reset_evt EventFd on reboot to
make sure that there is no spurious events in the EventFd.
Fixes: #783
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Now that vhost_user_backend and vm-virtio do support EVENT_IDX, use it
in vhost_user_block to reduce the number of notifications sent between
the driver and the device.
This is specially useful when using active polling on the virtqueue,
as it'll be implemented by a future patch.
This is a snapshot of kvm_stat while generating ~60K IOPS with fio on
the guest without EVENT_IDX:
Event Total %Total CurAvg/s
kvm_entry 393454 20.3 62494
kvm_exit 393446 20.3 62494
kvm_apic_accept_irq 378146 19.5 60268
kvm_msi_set_irq 369720 19.0 58881
kvm_fast_mmio 370497 19.1 58817
kvm_hv_timer_state 10197 0.5 1715
kvm_msr 8770 0.5 1443
kvm_wait_lapic_expire 7018 0.4 1118
kvm_apic 2768 0.1 538
kvm_pv_tlb_flush 2028 0.1 360
kvm_vcpu_wakeup 1453 0.1 278
kvm_apic_ipi 1384 0.1 269
kvm_fpu 1148 0.1 164
kvm_pio 574 0.0 82
kvm_userspace_exit 574 0.0 82
kvm_halt_poll_ns 24 0.0 3
And this is the snapshot while doing the same thing with EVENT_IDX:
Event Total %Total CurAvg/s
kvm_entry 35506 26.0 3873
kvm_exit 35499 26.0 3873
kvm_hv_timer_state 14740 10.8 1672
kvm_apic_accept_irq 13017 9.5 1438
kvm_msr 12845 9.4 1421
kvm_wait_lapic_expire 10422 7.6 1118
kvm_apic 3788 2.8 502
kvm_pv_tlb_flush 2708 2.0 340
kvm_vcpu_wakeup 1992 1.5 258
kvm_apic_ipi 1894 1.4 251
kvm_fpu 1476 1.1 164
kvm_pio 738 0.5 82
kvm_userspace_exit 738 0.5 82
kvm_msi_set_irq 701 0.5 69
kvm_fast_mmio 238 0.2 4
kvm_halt_poll_ns 50 0.0 1
kvm_ple_window_update 28 0.0 0
kvm_page_fault 4 0.0 0
It can be clearly appreciated how the number of vm exits per second,
specially the ones related to notifications (kvm_fast_mmio and
kvm_msi_set_irq) is drastically lower.
Signed-off-by: Sergio Lopez <slp@redhat.com>
Add helpers to Vring and VhostUserSlaveReqHandler for EVENT_IDX, so
consumers of this crate can make use of this feature.
Signed-off-by: Sergio Lopez <slp@redhat.com>
VIRTIO_RING_F_EVENT_IDX is a virtio feature that allows to avoid
device <-> driver notifications under some circunstances, most
notably when actively polling the queue.
This commit implements support for in in the vm-virtio
crate. Consumers of this crate will also need to add support for it by
exposing the feature and calling using update_avail_event() and
get_used_event() accordingly.
Signed-off-by: Sergio Lopez <slp@redhat.com>
Relying on the latest vm-memory version, including the freshly
introduced structure GuestMemoryAtomic, this patch replaces every
occurrence of Arc<ArcSwap<GuestMemoryMmap> with
GuestMemoryAtomic<GuestMemoryMmap>.
The point is to rely on the common RCU-like implementation from
vm-memory so that we don't have to do it from Cloud-Hypervisor.
Fixes#735
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The integration test test_memory_mergeable_on has been fairly unstable
for quite some time now. Because it can take some time for the VM to be
spawned and to be able to perform a correct measure of the PSS, this
commit simply increases the time before such measure is done.
This should return more accurate PSS results, which should help
stabilize the test.
Fixes#781
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that vhost_user_fs rust daemon supports virtiofs's dax mode, this adds
the two dax tests accordingly.
Signed-off-by: Liu Bo <bo.liu@linux.alibaba.com>
This adds the missing part of supporting virtiofs dax on the slave end,
that is, receiving a socket pair fd from the master end to set up a
communication channel for sending setupmapping & removemapping messages.
Signed-off-by: Liu Bo <bo.liu@linux.alibaba.com>
This introduces setupmapping and removemapping methods to server.rs,
passthrough.rs and filesystem.rs in order to support virtiofs dax mode
inside guest.
Since we don't really want the server.rs to know that it is dealing with
vhost-user specifically, this is making it more generic by adding a new
trait which has three functions map()/unmap()/sync() corresponding to
fs_slave_{map, unmap, sync}, server.rs will take anything that implements
the trait.
Signed-off-by: Liu Bo <bo.liu@linux.alibaba.com>
This introduces SlaveFsCacheReq which implements
VhostUserMasterReqHandler to handle map/unmap requests.
Signed-off-by: Liu Bo <bo.liu@linux.alibaba.com>
The unit tests are run from cargo test through multiple threads of the
same process. For this reason, all these threads share their file
descriptors (because that's how this works on Linux), which means that
any of them can close a file descriptor opened from another thread.
In the context of create_listener() and accept_connection() tests, they
can run concurrently and this generates some failure when the file
descriptor create_listener() is binding to is being closed from the
accept_connection() test.
In order to avoid such race condition, this patch simply removes the
part of the unit test performing an explicit and unsafe file descriptor
closure.
Fixes#759
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
If no socket is supplied when enabling "vhost_user=true" on "--disk"
follow the "exe" path in the /proc entry for this process and launch the
network backend (via the vmm_path field.)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The unit tests require some specific Linux capabilities and also to have
access to /dev/kvm device. This commit makes sure we enable only what's
necessary instead of blindly enable full priviliges with --privileged
option.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We need the host IPC for sharing eventfds with KVM, and the host network
for VFIO.
We also enforce the no-seccomp setting on the container, to overcome any
potential filtering set by our container's Ubuntu base.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
When a virtio-fs device is created with a dedicated shared region, by
default the region should be mapped as PROT_NONE so that no pages can be
faulted in.
It's only when the guest performs the mount of the virtiofs filesystem
that we can expect the VMM, on behalf of the backend, to perform some
new mappings in the reserved shared window, using PROT_READ and/or
PROT_WRITE.
Fixes#763
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
If no socket is supplied when enabling "vhost_user=true" on "--net"
follow the "exe" path in the /proc entry for this process and launch the
network backend (via the vmm_path field.)
Currently this only supports creating a new tap interface as the network
backend also only supports that.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
If the connection to the backend fails, sleep and retry up to a count.
This failure could happen because the backend is not yet ready to handle
the connection which is a common case when working with a spawned
backend.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
It is necessary to do this at the start of the VMM execution rather than
later as it must be done in the main thread in order to satisfy the
checks required by PTRACE_MODE_READ_FSCREDS (see proc(5) and
ptrace(2))
The alternative is to run as CAP_SYS_PTRACE but that has its
disadvantages.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This allows the VMM to explicitly shutdown devices as part of the VM
shutdown ahead of what Drop::drop() would do.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add a build-script to propagate the git commit hash to other crates at
compile time through environment variables, and display the hash along
with the '--version' option.
Fixes#729
Signed-off-by: Bo Chen <chen.bo@intel.com>
The placement of the explicit run of "test_vfio" meant it was run with
MMIO rather than PCI which meant it always failed.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Just updating the documentation regarding the pre-creation of a tap
interface supporting multiqueue, by adding what the cloud-hypervisor
--net parameter should look like.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By detecting if an existing tap interface supports multiqueue, we now
have the information to determine if the command line parameters
regarding the number of queues is correct.
Fixes#738
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to identify better the problem whenever an existing TAP is
provided, this patch enhances the error returned for an easier
diagnosis of the problem.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
test_vfio has been failing consistently on the CI so mark it with
a "#[ignore]" and then forceably build it again but ignore the build
result.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Implement the exit_event() method on the VhostUserBackend trait. It is
necessary to specify a custom exit event id in this case as the loop is
also used for handling activity on the tap file descriptors.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Implement the newly added exit_event() method on the VhostUserBackend
trait to allow the backend to provide an EventFD for triggering an exit
of the worker thread.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Rather than handling the KILL_EVENT in the event handler itself use the
newly added support in VhostUserBackend for providing a kill event
framework.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
All backends currently provide their own implementation for triggering
the worker thread to exit via an EventFd. Modify the VhostUserBackend
trait to allow a common implementation strategy that backends can use to
provide an EventFd (and optional id) that can be used to trigger the
worker to exit.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Check the return value from the worker thread by saving the thread
handle and waiting for it to return.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The main thread returns a Result with any errors from it. Although the
error from the join itself was being returned the real error from the
thread was being ignored so ensure that it is forwarded.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This reverts commit 4a1af7f63c.
This change erroneously ignored the return value for the result which
meant that requests to break out of the loop due to a kill event were
lost.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
All our tests must be run as root and thus the build directory is owned
by root after we run any of them.
Start another container to fix all permissions whenever we're done with
our tests.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
By default we will run as root inside the container, which means all the
build artifacts will be owned by root. That prevents us from properly
cleaning our build from an unprivileged host user.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
When running the docker container there is no interactivity needed so
don't pass "-ti" to "docker run"
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
To mitigate Azure slow disk IO, we mount /tmp on tmpfs.
This is a reproduction of our CI environment, as setup by the
Jenkinsfile.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
2020-02-07 17:33:30 +01:00
290 changed files with 59207 additions and 30502 deletions
- [Firecracker and crosvm](#firecracker-and-crosvm)
- [5. Community](#5-community)
- [Contribute](#contribute)
- [Join us](#join-us)
# 1. What is Cloud Hypervisor?
**This project is an experiment and should not be used with production workloads.**
Cloud Hypervisor is an open source Virtual Machine Monitor (VMM) that runs on top of [KVM](https://www.kernel.org/doc/Documentation/virtual/kvm/api.txt).
The project focuses on exclusively running modern, cloud workloads, on top of a limited set of hardware architectures and platforms.
Cloud workloads refers to those that are usually run by customers inside a cloud provider. For our purposes this means modern
Linux* distributions with most I/O handled by paravirtualised devices (i.e. virtio), no requirement for legacy devices and recent CPUs and KVM.
Cloud workloads refers to those that are usually run by customers inside a cloud provider. For our purposes this means modern operating systems with most I/O handled by paravirtualised devices (i.e. virtio), no requirement for legacy devices, and 64-bit CPUs.
Cloud Hypervisor is implemented in [Rust](https://www.rust-lang.org/) and is based on the [rust-vmm](https://github.com/rust-vmm) crates.
@@ -37,27 +36,24 @@ Cloud Hypervisor is implemented in [Rust](https://www.rust-lang.org/) and is bas
### High Level
* KVM and KVM only based
* Minimal emulation
* Low latency
* Low memory footprint
* Low complexity
* High performance
* Small attack surface
* 64-bit support only
* Build time configurable CPU, memory, PCI and NVDIMM hotplug
* Machine to machine migration
- KVM based
- Minimal emulation
- Low latency
- Low memory footprint
- Low complexity
- High performance
- Small attack surface
- 64-bit support only
- CPU, memory, PCI hotplug
- Machine to machine migration
### Architectures
`cloud-hypervisor` only supports the `x86-64`CPU architecture for now.
We're planning to add support for the `AArch64` architecture in the future.
Cloud Hypervisor supports the `x86-64`and `AArch64` architectures. There are some small differences in functionality between the two architectures (see [#1125](https://github.com/cloud-hypervisor/cloud-hypervisor/issues/1125)).
### Guest OS
*`64-bit Linux`
Support for *modern* 64-bit Windows guest is being evaluated.
Cloud Hypervisor supports `64-bit Linux` with support for _modern_ 64-bit Windows guests currently under development.
@@ -156,8 +171,7 @@ Multiple arguments can be given to the `--disk` parameter.
#### Building your kernel
`cloud-hypervisor` also supports direct kernel boot into a `vmlinux` ELF kernel
image. In order to support virtio-fs and virtio-iommu we have our own development branch. You are of course able to use your own kernel but these instructions will continue with the version that we develop and test against.
Cloud Hypervisor also supports direct kernel boot into a `vmlinux` ELF kernel or `bzImage`. In order to support virtio-fs and virtio-iommu we have our own development branch. You are of course able to use your own kernel but these instructions will continue with the version that we develop and test against.
Direct kernel boot to userspace should work with a rootfs from most distributions.
## Hot Plug
This [document](https://github.com/cloud-hypervisor/cloud-hypervisor/blob/master/docs/hotplug.md) details how to add devices to
a running VM. Currently only CPU hot plug is supported.
Cloud Hypervisor supports hotplug of CPUs, passthrough devices (VFIO), `virtio-{net,block,pmem,fs,vsock}` and memory resizing. This [document](https://github.com/cloud-hypervisor/cloud-hypervisor/blob/master/docs/hotplug.md) details how to add devices to
a running VM.
## Device Model
Follow this [documentation](https://github.com/cloud-hypervisor/cloud-hypervisor/blob/master/docs/device_model.md).
Details of the device model can be found in this [documentation](https://github.com/cloud-hypervisor/cloud-hypervisor/blob/master/docs/device_model.md).
## TODO
We are not tracking the `cloud-hypervisor` TODO list from a specific git tracked file but through
We are not tracking the Cloud Hypervisor TODO list from a specific git tracked file but through
In order to satisfy the design goal of having a high-performance, security-focused hypervisor the decision
was made to use the [Rust](https://www.rust-lang.org/) programming language.
The language's strong focus on memory and thread safety makes it an ideal candidate for implementing VMMs
The language's strong focus on memory and thread safety makes it an ideal candidate for implementing VMMs.
Instead of implementing the VMM components from scratch, `cloud-hypervisor` is importing the [rust-vmm](https://github.com/rust-vmm)
Instead of implementing the VMM components from scratch, Cloud Hypervisor is importing the [rust-vmm](https://github.com/rust-vmm)
crates, and sharing code and architecture together with other VMMs like e.g. Amazon's [Firecracker](https://firecracker-microvm.github.io/)
and Google's [crosvm](https://chromium.googlesource.com/chromiumos/platform/crosvm/).
`cloud-hypervisor` embraces the rust-vmm project goals, which is to be able to share and re-use
as many virtualization crates as possible. As such, the `cloud-hypervisor` relationship with the rust-vmm
Cloud Hypervisor embraces the rust-vmm project goals, which is to be able to share and re-use
as many virtualization crates as possible. As such, the Cloud Hypervisor relationship with the rust-vmm
project is twofold:
1. It will use as much of the rust-vmm code as possible. Any new rust-vmm crate that's relevant to the project
goals will be integrated as soon as possible.
2. As it is likely that the rust-vmm project will lack some of the features that `cloud-hypervisor` needs (e.g. ACPI,
VFIO, vhost-user, etc), we will be using the `cloud-hypervisor` VMM to implement and test them, and contribute them
2. As it is likely that the rust-vmm project will lack some of the features that Cloud Hypervisor needs (e.g. ACPI,
VFIO, vhost-user, etc), we will be using the Cloud Hypervisor VMM to implement and test them, and contribute them
back to the rust-vmm project.
## Firecracker and crosvm
A large part of the `cloud-hypervisor` code is based on either the Firecracker or the crosvm projects implementations.
A large part of the Cloud Hypervisor code is based on either the Firecracker or the crosvm projects implementations.
Both of these are VMMs written in Rust with a focus on safety and security, like Cloud Hypervisor.
However we want to emphasize that the Cloud Hypervisor project is neither a fork nor a reimplementation of any of those
@@ -287,6 +301,11 @@ crates as possible.
# 5. Community
The Cloud Hypervisor project follows the governance, and community guidelines described in
the [Community](https://github.com/cloud-hypervisor/community) repository.
## Contribute
We are working on building a global, diverse and collaborative community around the Cloud Hypervisor project.
Anyone who is interested in [contributing](CONTRIBUTING.md) to the project is welcome to participate.
@@ -298,27 +317,3 @@ etc, are all equal and welcome means of contribution. See the [CONTRIBUTING](CON
Get an [invite to our Slack channel](https://join.slack.com/t/cloud-hypervisor/shared_invite/enQtNjY3MTE3MDkwNDQ4LWQ1MTA1ZDVmODkwMWQ1MTRhYzk4ZGNlN2UwNTI3ZmFlODU0OTcwOWZjMTkwZDExYWE3YjFmNzgzY2FmNDAyMjI)
and [join us on Slack](https://cloud-hypervisor.slack.com/).
# 6. Security
**Reporting a Potential Security Vulnerability**: If you have discovered
potential security vulnerability in this project, please send an e-mail to
secure@intel.com. For issues related to Intel Products, please visit
https://security-center.intel.com.
It is important to include the following details:
- The projects and versions affected
- Detailed description of the vulnerability
- Information on known exploits
Vulnerability information is extremely sensitive. Please encrypt all security
vulnerability reports using our *PGP key*
A member of the Intel Product Security Team will review your e-mail and
contact you to to collaborate on resolving the issue. For more information on
how Intel works to resolve security issues, see: *Vulnerability Handling
If you want to build and test Cloud Hypervisor without having to install all the required dependencies, you can also turn to the development script: dev_cli.sh.
To build the development container:
```bash
./scripts/dev_cli.sh build-container
```
To build Cloud-hypervisor in the container:
```bash
./scripts/dev_cli.sh build
```
## Run
Assuming you have built Cloud-hypervisor with the development container, a VM can be started with command:
@@ -52,28 +52,28 @@ Direct kernel boot option is preferred since we need to provide the custom kerne
Because _vhost-user_ expects a dedicated process (__virtiofsd__ in this case) to be able to access the guest RAM to communicate through the _virtqueues_ with the driver running in the guest, `--memory` option needs to be slightly modified. It needs to specify a backing file for the memory so that an external process can access it.
Assuming you have `clear-kvm.img` and `custom-vmlinux.bin` on your system, here is the __cloud-hypervisor__ command you need to run:
Assuming you have `focal-server-cloudimg-amd64.raw` and `custom-vmlinux.bin` on your system, here is the __cloud-hypervisor__ command you need to run:
By default, DAX is enabled with a cache window of 8GiB. You can specify a custom size (let's say 4GiB for this example) for the cache by explicitly setting DAX and the cache size:
In case you don't want to use a shared window of cache to pass the shared files content, this means you will have to explicitly disable DAX with `dax=off`. Note that in this case, the `cache_size` parameter will be ignored.
@@ -5,9 +5,7 @@ Currently Cloud Hypervisor only support hot plugging of CPU devices.
## Kernel support
For hotplug on Cloud Hypervisor ACPI GED support is needed. This can either be achieved by turning on `CONFIG_ACPI_REDUCED_HARDWARE_ONLY`
or by using this kernel patch (available in 5.5rc1 and later): https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/patch/drivers/acpi/Makefile?id=ac36d37e943635fc072e9d4f47e40a48fbcdb3f0
This patch is integrated into the Clear Linux KVM and cloudguest images.
or by using this kernel patch (available in 5.5-rc1 and later): https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/patch/drivers/acpi/Makefile?id=ac36d37e943635fc072e9d4f47e40a48fbcdb3f0
## CPU Hot Plug
@@ -22,13 +20,14 @@ To use CPU hotplug start the VM with the number of max vCPUs greater than the nu
The extra vCPU threads will be created and advertised to the running kernel. The kernel does not bring up the CPUs immediately and instead the user must "online" them from inside the VM:
@@ -57,31 +56,36 @@ After a reboot the added CPUs will remain.
Removing CPUs works similarly by reducing the number in the "desired_vcpus" field of the reisze API. The CPUs will be automatically offlined inside the guest so there is no need to run any commands inside the guest:
As per adding CPUs to the guest, after a reboot the VM will be running with the reduced number of vCPUs.
## Memory Hot Plug
Extra memory can be added from a runing Cloud Hypervisor instance. This is controlled by two mechanisms:
### ACPI method
Extra memory can be added from a running Cloud Hypervisor instance. This is controlled by two mechanisms:
1. Allocating some of the guest physical address space for hotplug memory.
2. Making a HTTP API request to the VMM to ask for a new amount of RAM to be assigned to the VM. In the case of expanding the memory for the VM the new memory will be hotplugged into the running VM, if reducing the size of the memory then change will take effect after the next reboot.
To use memory hotplug start the VM specifying some size RAM in the "hotplug_size" parameter to the memory configuration. Not all the memory specified in this parameter will be available to hotplug as there are spacing and alignment requirements so it is recommended to make it larger than the hotplug RAM needed.
To use memory hotplug start the VM specifying some size RAM in the `hotplug_size` parameter to the memory configuration. Not all the memory specified in this parameter will be available to hotplug as there are spacing and alignment requirements so it is recommended to make it larger than the hotplug RAM needed.
Because the ACPI method is the default, there is no need to add the extra option `hotplug_method=acpi`.
The new memory is now available to use inside the VM:
@@ -110,4 +114,135 @@ Due to guest OS limitations is is necessary to ensure that amount of memory adde
The same API can also be used to reduce the desired RAM for a VM but the change will not be applied until the VM is rebooted.
Memory and CPU resizing can be combined together into the same HTTP API request.
Memory and CPU resizing can be combined together into the same HTTP API request.
### virtio-mem method
Extra memory can be added and removed from a running Cloud Hypervisor instance. This is controlled by two mechanisms:
1. Allocating some of the guest physical address space for hotplug memory.
2. Making a HTTP API request to the VMM to ask for a new amount of RAM to be assigned to the VM.
To use memory hotplug start the VM specifying some size RAM in the `hotplug_size` parameter along with `hotplug_method=virtio-mem` to the memory configuration.
The new memory is now available to use inside the VM:
```shell
free -h
total used free shared buff/cache available
Mem: 3.0Gi 71Mi 2.8Gi 0.0Ki 47Mi 2.8Gi
Swap: 32Mi 0B 32Mi
```
The same API can also be used to reduce the desired RAM for a VM. It is important to note that reducing RAM size might only partially work, as the guest might be using some of it.
## PCI Device Hot Plug
Extra PCI devices can be added and removed from a running Cloud Hypervisor instance. This is controlled by making a HTTP API request to the VMM to ask for the additional device to be added, or for the existing device to be removed.
To use PCI device hotplug start the VM with the HTTP server.
00:01.0 Unassigned class [ffff]: Red Hat, Inc. Virtio console (rev 01)
00:02.0 Mass storage controller: Red Hat, Inc. Virtio block device (rev 01)
00:03.0 Unassigned class [ffff]: Red Hat, Inc. Virtio RNG (rev 01)
```
After a reboot the added PCI device will remain.
### Remove PCI device
Removing a PCI device works the same way for all kind of PCI devices. The unique identifier related to the device must be provided. This identifier can be provided by the user when adding the new device, or by default Cloud Hypervisor will assign one.
* Developers who want to understand what log level to use and when,
* Users who want to debug issues with running their workloads in Cloud Hypervisor
## Control
The number of `-v` parameters passed to the `cloud-hypervisor` binary will determine the log level. Currenly the default is log messages up to `WARN:` (`warn!`) are included by default. The `--log-file` allows the log to be sent to a location other than `stderr`.
## Levels
### `error!()`
For immediate, unrecoverable errors where it does not make sense for the execution to continue as the behaviour of the VM is considerablely impacted.
Cloud Hypervisor should exit shortly after reporting this error (with a non-zero exit code). Generally this should be used during initial construction of the VM state before the virtual CPUs have begun running code.
A typical situation where this might occur is when the user is using command line options that conflict with each other or is trying to use a file that is not present on the filesystem.
Users should react to this error by checking their initial VM configuration.
### `warn!()`
A serious problem has occured but the execution of the VM can continue although some functionality might be impacted.
A typical example of where this level of message should be generated is during an API call request that cannot be fulfilled.
The user should investigate the meaning of this warning and take steps to ensure the correct functionality.
### `info!()`
Use `-v` to enable.
This level is for the benefit of developers. It should be used for sporadic and infrequent messages. The same message should not "spam" the logs. The VM should be usable when this level of debugging is enabled and trying to use `stdin/stdout` and the logs are going to `stderr`.
### `debug!()`
Use `-vv` to enable.
For the most verbose of logging messages. It is acceptable to "spam" the log with repeated invocations of the same message. This level of logging would be combined with `--log-file`.
Cloud Hypervisor supports using a MACVTAP device which is derived from a MACVLAN. Full details of configuring MACVLAN or MACVTAP is out of scope of this document. However the example below indicates how to bridge the guest directly onto the the network the host is on. Due to the lack of hairpin mode it not usually possible to reach the guest directly from the host.
```bash
# The MAC address must be attached to the macvtap and be used inside the guest
mac="c2:67:4f:53:29:cb"
# Host network adapter to bridge the guest onto
host_net="eno1"
# Create the macvtap0 as a new virtual MAC associated with the host network
sudo ip link add link "$host_net" name macvtap0 type macvtap
sudo ip link set macvtap0 address "$mac" up
sudo ip link show macvtap0
# A new character device is created for this interface
tapindex=$(< /sys/class/net/macvtap0/ifindex)
tapdevice="/dev/tap$tapindex"
# Ensure that we can access this device
sudo chown "$UID.$UID""$tapdevice"
# Use --net fd=3 to point to fd 3 which the shell has opened to point to the /dev/tapN device
target/debug/cloud-hypervisor \
--kernel ~/src/linux/vmlinux \
--disk path=~/workloads/focal.raw \
--cpus boot=1 --memory size=512M \
--cmdline "root=/dev/vda1 console=hvc0"\
--net fd=3,mac=$mac 3<>$"$tapdevice"
```
As the guest is now connected to the same L2 network as the host you can obtain an IP address based on your host network (potentially including via DHCP)
This option is mandatory when using the `--memory` parameter.
Value is an unsigned integer of 64 bits.
_Example_
```
--memory size=1G
```
### `mergeable`
Specifies if the pages from the guest RAM must be marked as _mergeable_. In
case this option is `true` or `on`, the pages will be marked with `madvise(2)`
to let the host kernel know which pages are eligible for being merged by the
KSM daemon.
This option can be used when trying to reach a higher density of VMs running
on a single host, as it will reduce the amount of memory consumed by each VM.
By default this option is turned off.
_Example_
```
--memory size=1G,mergeable=on
```
### `shared`
Specifies if the memory must be `mmap(2)` with `MAP_SHARED` flag.
By sharing a memory mapping, one can share the guest RAM with other processes
running on the host. One can use this option when running vhost-user devices
as part of the VM device model, as they will be driven by standalone daemons
needing access to the guest RAM content.
By default this option is turned off, which results in performing `mmap(2)`
with `MAP_PRIVATE` flag.
_Example_
```
--memory size=1G,shared=on
```
### `hugepages` and `hugepage_size`
Specifies if the memory must be created and `mmap(2)` with `MAP_HUGETLB` and size
flags. This performs a memory mapping relying on the specified huge page size. If no huge page size is supplied the system's default huge page size is used.
By using hugepages, one can improve the overall performance of the VM, assuming
the guest will allocate hugepages as well. Another interesting use case is VFIO
as it speeds up the VM's boot time since the amount of IOMMU mappings are
reduced.
The user is responsible for ensuring there are sufficient huge pages of the specified size for the VMM to use. Failure to do so may result in strange VMM behaviour.
By default this option is turned off.
_Example_
```
--memory size=1G,hugepages=on,hugepage_size=2M
```
### `hotplug_method`
Selects the way of adding and/or removing memory to/from a booted VM.
Possible values are `acpi` and `virtio-mem`. Default value is `acpi`.
_Example_
```
--memory size=1G,hotplug_method=acpi
```
### `hotplug_size`
Amount of memory that can be dynamically added to the VM.
Value is an unsigned integer of 64 bits. A value of 0 is invalid.
_Example_
```
--memory size=1G,hotplug_size=1G
```
### `hotplugged_size`
Amount of memory that will be dynamically added to the VM at boot. This option
allows for starting a VM with a certain amount of memory that can be reduced
during runtime.
This is only valid when the `hotplug_method` is `virtio-mem` as it does not
make sense for the `acpi` use case. When using ACPI, the memory can't be
resized after it has been extended.
This option is only valid when `hotplug_size` is specified, and its value can't
exceed the value of `hotplug_size`.
Value is an unsigned integer of 64 bits. A value of 0 is invalid.
`MemoryZoneConfig` or what is known as `--memory-zone` from the CLI perspective
is a power user parameter. It allows for a full description of the guest RAM,
describing how every memory region is backed and exposed to the guest.
```rust
structMemoryZoneConfig{
id: String,
size: u64,
file: Option<PathBuf>,
shared: bool,
hugepages: bool,
host_numa_node: Option<u32>,
hotplug_size: Option<u64>,
hotplugged_size: Option<u64>,
}
```
```
--memory-zone <memory-zone> User defined memory zone parameters "size=<guest_memory_region_size>,file=<backing_file>,shared=on|off,hugepages=on|off,host_numa_node=<node_id>,id=<zone_identifier>,hotplug_size=<hotpluggable_memory_size>,hotplugged_size=<hotplugged_memory_size>"
```
This parameter expects one or more occurences, allowing for a list of memory
zones to be defined. It must be used with `--memory size=0`, clearly indicating
that the memory will be described through advanced parameters.
Each zone is given a list of options which we detail through the following
sections.
### `id`
Memory zone identifier. This identifier must be unique, otherwise an error will
be returned.
This option is useful when referring to a memory zone previously created. In
particular, the `--numa` parameter can associate a memory zone to a specific
NUMA node based on the memory zone identifier.
This option is mandatory when using the `--memory-zone` parameter.
Value is a string.
_Example_
```
--memory size=0
--memory-zone id=mem0,size=1G
```
### `size`
Size of the memory zone.
This option is mandatory when using the `--memory-zone` parameter.
Value is an unsigned integer of 64 bits.
_Example_
```
--memory size=0
--memory-zone id=mem0,size=1G
```
### `file`
Path to the file backing the memory zone. This can be either a file or a
directory. In case of a file, it will be opened and used as the backing file
for the `mmap(2)` operation. In case of a directory, a temporary file with no
hard link on the filesystem will be created. This file will be used as the
backing file for the `mmap(2)` operation.
This option can be particularly useful when trying to back a part of the guest
RAM with a well known file. In the context of the snapshot/restore feature, and
if the provided path is a file, the snapshot operation will not perform any
copy of the guest RAM content for this specific memory zone since the user has
access to it and it would duplicate data already stored on the current
filesystem.
Value is a string.
_Example_
```
--memory size=0
--memory-zone id=mem0,size=1G,file=/foo/bar
```
### `shared`
Specifies if the memory zone must be `mmap(2)` with `MAP_SHARED` flag.
By sharing a memory zone mapping, one can share part of the guest RAM with
other processes running on the host. One can use this option when running
vhost-user devices as part of the VM device model, as they will be driven
by standalone daemons needing access to the guest RAM content.
By default this option is turned off, which result in performing `mmap(2)`
with `MAP_PRIVATE` flag.
_Example_
```
--memory size=0
--memory-zone id=mem0,size=1G,shared=on
```
### `hugepages`
Specifies if the memory zone must be `mmap(2)` with `MAP_HUGETLB` and
`MAP_HUGE_2MB` flags. This performs a memory zone mapping relying on 2MiB
pages instead of the default 4kiB pages.
By using hugepages, one can improve the overall performance of the VM, assuming
the guest will allocate hugepages as well. Another interesting use case is VFIO
as it speeds up the VM's boot time since the amount of IOMMU mappings are
reduced.
By default this option is turned off.
_Example_
```
--memory size=0
--memory-zone id=mem0,size=1G,hugepages=on
```
### `host_numa_node`
Node identifier of a node present on the host. This option will let the user
pick a specific NUMA node from which the memory must be allocated. After the
memory zone is `mmap(2)`, the NUMA policy for this memory mapping will be
applied through `mbind(2)`, relying on the provided node identifier. If the
node does not exist on the host, the call to `mbind(2)` will fail.
This option is useful when trying to back a VM memory with a specific type of
memory from the host. Assuming a host has two types of memory, with one slower
than the other, each related to a distinct NUMA node, one could create a VM
with slower memory accesses by backing the entire guest RAM from the furthest
NUMA node on the host.
This option also gives the opportunity to create a VM with non uniform memory
accesses as one could define a first memory zone backed by fast memory, and a
second memory zone backed by slow memory.
Value is an unsigned integer of 32 bits.
_Example_
```
--memory size=0
--memory-zone id=mem0,size=1G,host_numa_node=0
```
### `hotplug_size`
Amount of memory that can be dynamically added to the memory zone. Since
`virtio-mem` is the only way of resizing a memory zone, one must specify
the `hotplug_method=virtio-mem` to the `--memory` parameter.
Value is an unsigned integer of 64 bits. A value of 0 is invalid.
_Example_
```
--memory size=0,hotplug_method=virtio-mem
--memory-zone id=mem0,size=1G,hotplug_size=1G
```
### `hotplugged_size`
Amount of memory that will be dynamically added to a memory zone at VM's boot.
This option allows for starting a VM with a certain amount of memory that can
be reduced during runtime.
This is only valid when the `hotplug_method` is `virtio-mem` as it does not
make sense for the `acpi` use case. When using ACPI, the memory can't be
resized after it has been extended.
This option is only valid when `hotplug_size` is specified, and its value can't
exceed the value of `hotplug_size`.
Value is an unsigned integer of 64 bits. A value of 0 is invalid.
`NumaConfig` or what is known as `--numa` from the CLI perspective has been
introduced to define a guest NUMA topology. It allows for a fine description
about the CPUs and memory ranges associated with each NUMA node. Additionally
it allows for specifying the distance between each NUMA node.
```rust
structNumaConfig{
id: u32,
cpus: Option<Vec<u8>>,
distances: Option<Vec<NumaDistance>>,
memory_zones: Option<Vec<String>>,
}
```
```
--numa <numa> Settings related to a given NUMA node "id=<node_id>,cpus=<cpus_id>,distances=<list_of_distances_to_destination_nodes>,memory_zones=<list_of_memory_zones>"
```
### `guest_numa_id`
Node identifier of a guest NUMA node. This identifier must be unique, otherwise
an error will be returned.
This option is mandatory when using the `--numa` parameter.
Value is an unsigned integer of 32 bits.
_Example_
```
--numa guest_numa_id=0
```
### `cpus`
List of virtual CPUs attached to the guest NUMA node identified by the
`guest_numa_id` option. This allows for describing a list of CPUs which
must be seen by the guest as belonging to the NUMA node `guest_numa_id`.
One can use this option for a fine grained description of the NUMA topology
regarding the CPUs associated with it, which might help the guest run more
efficiently.
Multiple values can be provided to define the list. Each value is an unsigned
integer of 8 bits.
For instance, if one needs to attach all CPUs from 0 to 4 to a specific node,
the syntax using `-` will help define a contiguous range with `cpus=0-4`. The
same example could also be described with `cpus=0:1:2:3:4`.
A combination of both `-` and `:` separators is useful when one might need to
describe a list containing all CPUs from 0 to 99 and the CPU 255, as it could
simply be described with `cpus=0-99:255`.
_Example_
```
--cpus boot=8
--numa guest_numa_id=0,cpus=1-3:7
--numa guest_numa_id=1,cpus=0:4-6
```
### `distances`
List of distances between the current NUMA node referred by `guest_numa_id`
and the destination NUMA nodes listed along with distances. This option let
the user choose the distances between guest NUMA nodes. This is important to
provide an accurate description of the way non uniform memory accesses will
perform in the guest.
One or more tuple of two values must be provided through this option. The first
value is an unsigned integer of 32 bits as it represents the destination NUMA
node. The second value is an unsigned integer of 8 bits as it represents the
distance between the current NUMA node and the destination NUMA node. The two
values are separated by `@` (`value1@value2`), meaning the destination NUMA
node `value1` is located at a distance of `value2`. Each tuple is separated
from the others with `:` separator.
For instance, if one wants to define 3 NUMA nodes, with each node located at
different distances, it can be described with the following example.
_Example_
```
--numa guest_numa_id=0,distances=1@15:2@25
--numa guest_numa_id=1,distances=0@15:2@20
--numa guest_numa_id=2,distances=0@25:1@20
```
### `memory_zones`
List of memory zones attached to the guest NUMA node identified by the
`guest_numa_id` option. This allows for describing a list of memory ranges
which must be seen by the guest as belonging to the NUMA node `guest_numa_id`.
This option can be very useful and powerful when combined with `host_numa_node`
option from `--memory-zone` parameter as it allows for creating a VM with non
uniform memory accesses, and let the guest know about it. It allows for
exposing memory zones through different NUMA nodes, which can help the guest
workload run more efficiently.
Multiple values can be provided to define the list. Each value is a string
referring to an existing memory zone identifier. Values are separated from
cloud-hypervisor can emulate one or more virtual network interfaces, represented at the hypervisor host by [tap devices](https://www.kernel.org/doc/Documentation/networking/tuntap.txt). This guide briefly describes, in a manual and distribution neutral way, how to setup and use networking with cloud-hypevisor.
cloud-hypervisor can emulate one or more virtual network interfaces, represented at the hypervisor host by [tap devices](https://www.kernel.org/doc/Documentation/networking/tuntap.txt). This guide briefly describes, in a manual and distribution neutral way, how to setup and use networking with cloud-hypervisor.
## Multiple queue support for net devices ##
@@ -14,7 +14,7 @@ Note:
- Currently, it does not support to use ethtool to change the combined queue numbers in guest.
- Multiple queue is enabled for vhost-user-net backend in cloud-hypervisor, however, multiple thread is not added to handle mq, thus, the performance for vhost-user-net backend is not supposed to be improved. The multiple thread will be added for backend later.
- Performance test for vhost-user-net will be covered once vhost-user-net backend has mulitple thread supported.
- Performance test for vhost-user-net will be covered once vhost-user-net backend has multiple thread supported.
- Performance test for virtio-net is done by comparing 2 queue pairs with 1 queue pairs, that to run 2 iperf3 sessions in the same test environments, throughput is improved about 37%.
## Start cloud-hypervisor with net devices
@@ -25,9 +25,9 @@ Use one `--net` command-line argument from cloud-hypervisor to specify the emula
@@ -51,6 +51,11 @@ If the tap device is pre-created on host before guest boot up. To use multiple q
[root@localhost ~]# ip tuntap add name ich0 mode tap multi_queue
```
And the `--net` device should specify support for multiple queues. `num_queues` must be a multiple of 2 starting at least from 4 since multiple queues really means multiple queue pairs. We need at least 2 pairs for this configuration to be correct:
`perf` can be used to profile the `cloud-hypervisor` binary but it is necessary to make some modifications to the the build in order to produce a binary that gives useful results.
## Building a suitable binary
Modify the `Cargo.toml` file to add `debug = 1` to the `[profile.release]` block. It should look like this:
```
[profile.release]
lto = true
debug = 1
```
This adds the symbol information to the release binary but does not otherwise affect the performance.
The binary must also be built with frame pointers included so that the call graph can be captured by the profiler.
$ perf record -g target/release/cloud-hypervisor \
--kernel ~/src/linux/vmlinux \
--pmem file=~/workloads/focal.raw \
--cpus boot=1 --memory size=1G \
--cmdline "root=/dev/pmem0p1 console=ttyS0" \
--serial tty --console off \
--api-socket=/tmp/api1
```
For analysing the samples:
```
$ perf report -g
```
If profiling with a network device attached either the TAP device must be already created and configured or the profiling must be done as root so that the TAP device can be created.
Cloud Hypervisor supports UEFI boot through the utilization of the EDK II based UEFI firmware.
## Building UEFI Firmware
To avoid any unnecessary issues, it is recommended to use Ubuntu 18.04 and its default toolset. Any other compatible Linux distribution is otherwise suitable, however it is suggested to use a temporary Docker container with Ubuntu 18.04 for a quick build on an existing Linux machine.
The commands below will compile an OVMF firmware suitable for Cloud Hypervisor.
After the successful build, the resulting firmware binaries are available under `Build/OvmfCh/DEBUG_GCC5/FV` underneath the edk2 checkout.
## Using OVMF Binaries
Any UEFI capable image can be booted using the Cloud Hypervisor specific firmware. Windows guests under Cloud Hypervisor only support UEFI boot, therefore OVMF is mandatory there.
To make Cloud Hypervisor use UEFI boot, pass the `OVMF.fd` file path as an argument to the `--kernel` option. The firmware file will be opened in read only mode.
The same firmware can be used with Cloud Hypervisor or with QEMU. This is particularly useful if using QEMU for the preparation phase.
## Building UEFI Firmware with Compatibility Support Module (CSM)
CSM is a module that allows to boot legacy operating systems using the OVMF firmware. OVMF can embed a CSM build of SeaBIOS. To build the SeaBIOS with CSM support, add `CONFIG_CSM=y` to `.config` before the build. The outcome `out/Csm16.bin` is to be moved into `OvmfPkg/Csm/Csm16/Csm16.bin` before OVMF is built. Then, the OVMF build will have to be passed the `-D CSM_ENABLE` option in order to generate a legacy aware UEFI firmware. At the current stage, all the necessary patches are included in the Cloud Hypervisor specific [SeaBIOS branch](https://github.com/cloud-hypervisor/seabios/tree/ch). Taking into account the previous instructions, the modified command sequence to compile an OVMF binary with CSM support is the following one:
Please note, that the CSM support has currently only been tested with Linux guests. There are no plans to provide legacy support for other OSes (e.g. Windows).
The test runs with multiple queue (MQ) support enabled, using 2 pairs of TX/RX queues defined for both OVS and the virtual machine. Here are the details on how the test can be run.
_Setup OVS_
`ovs_test.sh` is created to setup and start OVS. OVS will provide the `dpdkvhostuser` backend running in server mode.
Starting with the release version [0.10.0](https://github.com/cloud-hypervisor/cloud-hypervisor/releases/tag/v0.10.0), Cloud Hypervisor supports Windows guests.
__Requirements__
- Host with KVM enabled
- [UEFI](uefi.md) capable Windows guest image with Virtio drivers integrated
Any modern Windows Server version is compatible. Cloud Hypervisor has been successfully tested with Windows Server 2019 and Windows Server Core 2004.
At the current stage, only UEFI capable Windows images are supported. This implies the presence of the OVMF firmware during the Windows installation and in any subsequent usage. BIOS boot is not supported.
The subsequent sections will tell, in detail, how to prepare an appropriate Windows image.
## Image Preparation
### Installation using the stock Windows ISO
__Prerequisites__
- QEMU, version >=5.0.0 is recommended.
- Windows installation ISO. Obtained through MSDN, Visual Studio subscription, evaluation center, etc.
- With the suggested image size of 30G, there should be enough free disk space to hold the installation ISO and any other necessary files
This step currently requires QEMU to install Windows onto the guest. QEMU is only used at the preparation stage, the resulting image is then fully functional with Cloud Hypervisor.
Preparing several command parts as these will be used in the follow up sections as well.
Before the installation can proceed, point the Windows installation program to the VirtIO disk and install the necessary storage controller drivers. After that, the attached hard drive will become visible and the actual installation can commence.
After the installation has completed, proceed further to the configuration section. QEMU will be needed at least once more to enable the Windows Special Administration Console (SAC) and to possibly install extra device drivers.
## Image Usage
The basic command to boot a Windows image. The configuration section should be checked before executing it for the first time.
```shell
cloud-hypervisor \
--kernel ./$OVMF_DIR/OVMF.fd \
--disk path=./$IMG_FILE\
--cpus boot=1,kvm_hyperv=on \
--memory size=4G \
--serial tty \
--console off \
--net tap=
```
It is necessary to always:
- Carry the OVMF firmware in the `--kernel` option
- Add `kvm_hyperv=on` to the `--cpus` option
In cases where the host processor supports address space > 39 bits, it might be necessary to limit the address space. It can be done by appending the option `max_phys_bits=X` to the `--cpus` parameter, where `X` is the number of bits to be supported. Windows was tested to support at least 39-bit address space.
To daemonize the Cloud Hypervisor process, `nohup` can be used. Some STDIO redirections might need to be done. In a simple case it is sufficient to just redirect all the output to `/dev/null`.
## Image Configuration
### Device Drivers
After the Windows installation has finished under QEMU, there might be still devices with no drivers installed. This might happen for example, when a device was not used during the installation. In particular it is important to ensure that the VirtIO network device is setup correctly because further steps for the configuration and the usage require network in most case.
Boot once more under QEMU and use the [Device Manager](https://support.microsoft.com/en-in/help/4028443/windows-10-update-drivers), to ensure all the device drivers, and especially the network card, are installed correctly. Also, as Cloud Hypervisor can introduce new devices, it is advisable to repeat the procedure while booted under Cloud Hypervisor, when the RDP access to the image is functional.
### Windows Special Administration Console (SAC) enablement
SAC provides a text based console access to the Windows guest. As Cloud Hypervisor doesn't implement a VGA adaptor, SAC is an important instrument for the Windows guest management.
Boot the Windows image under QEMU and execute the below commands to permanently enable SAC
```cmd
bcdedit /emssettings emsport:1 emsbaudrate:115200
bcdedit /ems on
bcdedit /bootems on
```
Once SAC is enabled, the image can be booted under Cloud Hypervisor. The SAC prompt will show up
<pre>
Computer is booting, SAC started and initialized.
Use the "ch -?" command for information about using channels.
Use the "?" command for general help.
SAC>
</pre>
To open a console on the guest, the command sequence below can be used
<pre>
SAC>cmd
The Command Prompt session was successfully launched.
SAC>
EVENT: A new channel has been created. Use "ch -?" for channel help.
Channel: Cmd0001
SAC>ch -si 1
</pre>
See also the [links](#Links) section for a more extended SAC documentation.
## Network
This section illustrates the Windows specific corner points for the VM network configuration. For the extended networking guide, including bridging for multiple VMs, follow [networking.md](networking.md).
### Basic Networking
As the simplest option, using `--net tap=` in the Cloud Hypervisor command line will create a `vmtapX` device on the host with the default IPv4 adress `192.168.249.1`. After SAC becomes available, the guest configuration can be set with
This allows for SSH login from a remote machine, for example through the `administrator` user: `ssh administrator@192.168.249.2`. For a more detailed OpenSSH guide, please follow the MSDN article from the [links](#links) section.
## Debugging
The Windows guest debugging process relies heavily on QEMU and [socat](http://www.dest-unreach.org/socat/). The procedure requires two Windows VMs:
- A debugger VM running under QEMU.
- A debuggee, a Windows VM that has been created in the previous steps, running under Cloud Hypervisor or QEMU.
The connection between both guests happens over TCP, whereby on the guest side it is automatically translated to a COM port. Because the VMs are connected through TCP, the debugging infrastructure can be distributed over the network. The serial port, while slowly transferring data, is common enough to support a wide range of cases and tools.
In this excercise, [WinDbg](https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/) is used. Any other debugger of choice with the ability to use serial connection can be used instead.
### Debugger and Debuggee
#### WinDbg VM
For simplicity, the debugger VM is supposed to be only running under QEMU. It will require VGA and doesn't neccessarily depend on UEFI. As an OS, it can carry any supported Windows OS where the debugger of choice can be installed. The simplest way is to follow the image preparation instructions from the previous chapter, but avoid using the OVMF firmware. It is also not required to use VirtIO drivers, whereby it might be useful in some case. Though, while creating the image file for the debugger VM, be sure to choose a sufficient disk size that counts in the need to save the corresponding debug symbols and sources.
To create the debugger Windows VM, the following command can be used:
A non server Windows OS like Windows 10 can be used to carry the debugging tools in the debugger VM.
#### Debuggee VM
The debuggee VM is the one that we've learned to configure and run in the first section. There might be various reasons to debug. For example, there could be an issue in the Windows guest with an emulated device or an included driver. Or, we might want to develop a custom feature like a kernel driver to be available in the guest.
Note, that there are several ways to debug Windows, not all of them need to be enabled at the same time. For example, if developing a kernel module, the only useful options would be to configure for the serial debugging and enable the kernel debug. In that case, any crash or misbehavior in the boot loader or kernel would be ignored. The commands below must be run as administrator on the debuggee guest VM.
##### Turn On Serial Debugging
This will configure the debugging to be enabled and instruct to use the serial port for it.
```cmd
bcdedit /dbgsettings serial debugport:1 baudrate:115200
```
##### Turn On Kernel Debuging
```cmd
bcdedit /debug on
```
##### Turn On Boot Loader Debug
```cmd
bcdedit /bootdebug on
```
##### Turn on boot manager debug
```cmd
bcdedit /set {bootmgr} bootdebug on
```
##### Disable Recovery Screen On Boot Failure
There could be a situation, where a crash is debugged. In such cases, the guest could be left in an inconsistent state. The default Windows behavior would be to boot into the recovery screen, however in some cases it might be not desired. To make Windows ignore failures and always proceed to booting the OS, use the command below:
Note, this VM has the networking enabled. It is needed, because symbols and sources might need to be fetched from a network location.
Also, notice the `-serial` parameter - that's what does the magic on exposing the serial port to the guest while connecting the debugger VM with a client VM through the network. SAC/EMS needs to be disabled in the debugger VM, as otherwise the COM device might be blocked.
Hereafter, WinDbg can be started using a command below:
```cmd
set_NT_DEBUG_PORT=com1
set_NT_DEBUG_BAUD_RATE=115200
windbg -v -d -k
```
Once started, WinDbg will wait for an incoming connection which is going to be initialized by the debuggee VM started in the next section.
#### Invoke the Debuggee VM
##### Under QEMU
Essentially it would be the command like depicted in the guest preparation sections, with a few modifications:
It is to see, that `-serial` parameter is used here, to establish the connection with the debugger VM.
To disable HPET, attach `--no-hpet`. To enable hypervisor reference timer, use `-cpu host,hv-time`. These and other options can be used to achieve better [Hyper-V compatibility](https://archive.fosdem.org/2019/schedule/event/vai_enlightening_kvm/attachments/slides/2860/export/events/attachments/vai_enlightening_kvm/slides/2860/vkuznets_fosdem2019_enlightening_kvm.pdf).
##### Cloud Hypervisor
The `socat` tool is used to establish the QEMU compatible behavior. Here as well, the Cloud Hypervisor command used to run the Windows guest is to be used. Put the command into a shell script:
The reason to pack the command into the shell script is that the command might contain a comma. When using SYSTEM, the shell command can't contain `,` or `!!`.
## Links
- [Fedora VirtIO guide for Windows](https://docs.fedoraproject.org/en-US/quick-docs/creating-windows-virtual-machines-using-virtio-drivers/)
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
