DescriptorChain::is_valid() wrongly used .checked_offset() to attempt to
validate that the descriptor's data is in valid memory. This works in
all cases except where the guest has placed the data at the very end of
the guest memory as the offset + offset will be outside the range (as
the combined offset will be the next byte and as such out of the guest
memory). Instead use the function .check_range() takes an offset and a
length to validate
This fixes issues see with error messages featuring the
DescriptorChainTooShort error.
Fixes: #2424
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
(cherry picked from commit 1eb4ebe3d1)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
There are two parts:
- Unconditionally zero the output area. The length of the incoming
vector has been seen from 1 to 4 bytes, even though just the first
byte might need to be handled. But also, this ensures any possibly
unhandled offset will return zeroed result to the caller. The former
implementation used an I/O port which seems to behave differently from
MMIO and wouldn't require explicit output zeroing.
- An access with zero offset still takes place and needs to be handled.
Fixes#2437.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
(cherry picked from commit 9e9aba7c0b)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The following is from the Hyper-V specification v6.0b.
Cpuid leaf 0x40000003 EDX:
Bit 3: Support for physical CPU dynamic partitioning events is
available.
When Windows determines to be running under a hypervisor, it will
require this cpuid bit to be set to support dynamic CPU operations.
Cpuid leaf 0x40000004 EAX:
Bit 5: Recommend using relaxed timing for this partition. If
used, the VM should disable any watchdog timeouts that
rely on the timely delivery of external interrupts.
This bit has been figured out as required after seeing guest BSOD
when CPU hotplug bit is enabled. Race conditions seem to arise after a
hotplug operation, when a system watchdog has expired.
Closes#1799.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
(cherry picked from commit 5b168f54a6)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
error: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
Error: --> devices/src/legacy/rtc_pl031.rs:73:1
|
73 | impl Into<libc::clockid_t> for ClockType {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= note: `-D clippy::from-over-into` implied by `-D warnings`
= help: consider to implement `From` instead
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
error: redundant slicing of the whole range
Error: --> devices/src/legacy/gpio_pl061.rs:298:37
|
298 | let value = read_le_u32(&data[..]);
| ^^^^^^^^^ help: use the original slice instead: `data`
|
= note: `-D clippy::redundant-slicing` implied by `-D warnings`
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#redundant_slicing
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
error: name `GPIOInterruptDisabled` contains a capitalized acronym
Error: --> devices/src/legacy/gpio_pl061.rs:46:5
|
46 | GPIOInterruptDisabled,
| ^^^^^^^^^^^^^^^^^^^^^ help: consider making the acronym lowercase, except the initial letter: `GpioInterruptDisabled`
|
= note: `-D clippy::upper-case-acronyms` implied by `-D warnings`
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#upper_case_acronyms
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
warning: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
--> hypervisor/src/vm.rs:41:1
|
41 | impl Into<u64> for DataMatch {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= note: `#[warn(clippy::from_over_into)]` on by default
= help: consider to implement `From` instead
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into
warning: 1 warning emitted
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
warning: name `AcpiPMTimerDevice` contains a capitalized acronym
--> devices/src/acpi.rs:175:12
|
175 | pub struct AcpiPMTimerDevice {
| ^^^^^^^^^^^^^^^^^ help: consider making the acronym lowercase, except the initial letter: `AcpiPmTimerDevice`
|
= note: `#[warn(clippy::upper_case_acronyms)]` on by default
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#upper_case_acronyms
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
warning: name `IORegion` contains a capitalized acronym
--> pci/src/configuration.rs:320:5
|
320 | IORegion = 0x01,
| ^^^^^^^^ help: consider making the acronym lowercase, except the initial letter (notice the capitalization): `IoRegion`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#upper_case_acronyms
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
warning: name `ConvertFromUTF8` contains a capitalized acronym
--> virtio-devices/src/vsock/unix/mod.rs:32:5
|
32 | ConvertFromUTF8(std::str::Utf8Error),
| ^^^^^^^^^^^^^^^ help: consider making the acronym lowercase, except the initial letter: `ConvertFromUtf8`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#upper_case_acronyms
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
warning: name `InvalidCPUCount` contains a capitalized acronym
--> src/bin/ch-remote.rs:24:5
|
24 | InvalidCPUCount(std::num::ParseIntError),
| ^^^^^^^^^^^^^^^ help: consider making the acronym lowercase, except the initial letter: `InvalidCpuCount`
|
= note: `#[warn(clippy::upper_case_acronyms)]` on by default
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#upper_case_acronyms
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
warning: name `TranslateGVA` contains a capitalized acronym
--> hypervisor/src/arch/emulator/mod.rs:51:5
|
51 | TranslateGVA(#[source] anyhow::Error),
| ^^^^^^^^^^^^ help: consider making the acronym lowercase, except the initial letter: `TranslateGva`
|
= note: `#[warn(clippy::upper_case_acronyms)]` on by default
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#upper_case_acronyms
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
error: redundant slicing of the whole range
--> vhost_user_block/src/lib.rs:396:31
|
396 | right.copy_from_slice(&data[..]);
| ^^^^^^^^^ help: use the original slice instead: `data`
|
= note: `-D clippy::redundant-slicing` implied by `-D warnings`
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#redundant_slicing
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
error: name `RSDPPastRamEnd` contains a capitalized acronym
--> arch/src/lib.rs:59:5
|
59 | RSDPPastRamEnd,
| ^^^^^^^^^^^^^^ help: consider making the acronym lowercase, except the initial letter: `RsdpPastRamEnd`
|
= note: `-D clippy::upper-case-acronyms` implied by `-D warnings`
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#upper_case_acronyms
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
error: writing `&PathBuf` instead of `&Path` involves a new object where a slice will do.
--> block_util/src/lib.rs:68:31
|
68 | fn build_device_id(disk_path: &PathBuf) -> result::Result<String, Error> {
| ^^^^^^^^ help: change this to: `&Path`
|
= note: `-D clippy::ptr-arg` implied by `-D warnings`
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#ptr_arg
error: writing `&PathBuf` instead of `&Path` involves a new object where a slice will do.
--> block_util/src/lib.rs:83:39
|
83 | pub fn build_disk_image_id(disk_path: &PathBuf) -> Vec<u8> {
| ^^^^^^^^ help: change this to: `&Path`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#ptr_arg
error: writing `&PathBuf` instead of `&Path` involves a new object where a slice will do.
--> block_util/src/lib.rs:68:31
|
68 | fn build_device_id(disk_path: &PathBuf) -> result::Result<String, Error> {
| ^^^^^^^^ help: change this to: `&Path`
|
= note: `-D clippy::ptr-arg` implied by `-D warnings`
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#ptr_arg
error: writing `&PathBuf` instead of `&Path` involves a new object where a slice will do.
--> block_util/src/lib.rs:83:39
|
83 | pub fn build_disk_image_id(disk_path: &PathBuf) -> Vec<u8> {
| ^^^^^^^^ help: change this to: `&Path`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#ptr_arg
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
error: redundant slicing of the whole range
--> net_util/src/mac.rs:60:35
|
60 | bytes[..].copy_from_slice(&src[..]);
| ^^^^^^^^ help: use the original slice instead: `src`
|
= note: `-D clippy::redundant-slicing` implied by `-D warnings`
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#redundant_slicing
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
error: name `TYPE_UNKNOWN` contains a capitalized acronym
--> vm-virtio/src/lib.rs:48:5
|
48 | TYPE_UNKNOWN = 0xFF,
| ^^^^^^^^^^^^ help: consider making the acronym lowercase, except the initial letter: `Type_Unknown`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#upper_case_acronyms
error: name `GetDeviceID` contains a capitalized acronym
--> block_util/src/lib.rs:138:5
|
138 | GetDeviceID,
| ^^^^^^^^^^^ help: consider making the acronym lowercase, except the initial letter: `GetDeviceId`
|
= note: `-D clippy::upper-case-acronyms` implied by `-D warnings`
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#upper_case_acronyms
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
With CONFIG_PVH in stable kernels for some time we should deprecate the
use of alternative boot methods since this will lead to a much simpler
boot flow and CI process.
See: #2231
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
There are multiple reports of DescriptorChainTooShort errors and so add
some extra debugging to aid the debugging of this issue.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Drop the generic type E and use IrqRoutngEntry directly. This allows
dropping a bunch of trait bounds from code.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Their make_entry functions look the same now. Extract the logic to a
common function.
No functional change.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
This returns trues if this descriptor has another descriptor linked to
it. Not whether this descriptor chain has another one following it.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The 0.3.0 release (re)supports booting stock Ubuntu images so users
should be pointed towards that.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
There is no point in queueing an empty descriptor in the list of iovecs.
Let's simply ignore such case and avoid some unnecessary processing.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The hotplug documentation was missing the explanation on how to use
memory hotplug with virtio-mem.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Document that seccomp is on and how to disable it along with a pointer
on how to identify missing syscalls during development.
Fixes#993
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
It must be specified as excluded from the workspace as it must not be
built on non-test targets due to issues with the ssh2 dependency and the
musl toolchain.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
refactor vec_with_array_field to common hypervisor code
so that mshv can also make use of it.
Signed-off-by: Vineeth Pillai <viremana@linux.microsoft.com>
Bumps iced-x86 from 1.10.3 to 1.11.0.
Manual update of the code was needed since memory_displacement() was
deprecated and replaced with either memory_displacement32() or
memory_displacement64().
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
There's no need to have the code creating the passthrough_device being
duplicated since we can factorize it in a function used in both cases
(both cold plugged and hot plugged devices VFIO devices).
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Extend and use the existing DeviceTree to retrieve useful information
related to PCI devices. This removes the duplication with pci_devices
field which was internal to the DeviceManager.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Make the code a bit clearer by changing the naming of the structure
holding the list of IRQs reserved for PCI devices. It is also modified
into an array of 32 entries since we know this is the amount of PCI
slots that is supported.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We define a new enum in order to classify PCI device under virtio or
VFIO. This is a cleaner approach than using the Any trait, and
downcasting it to find the object back.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Introduces a tuple holding both information needed by pci_id_list and
pci_devices.
Changes pci_devices to be a BTreeMap of this new tuple.
Now that pci_devices holds the information needed from pci_id_list,
pci_id_list is no longer needed.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In anticipation for further factorization, the pci_id_list is now a
hashmap of PCI b/d/f leading to each device name.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Instead of relying on a PCI specific device list, we use the DeviceTree
as a reference to determine if a device name is already in use or not.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Remove the startup message containing incomplete VM configuration
details. It's a bit unusual for a tool such as Cloud Hypervisor to print
those kind of details which are a direct representation of the command
line.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This includes:
* OS disk image management
* Cloud init creation
* SSH to guest access
* Waiting for guest to boot
This will be useful in other projects that want to do similar things in
their integration tests.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Only if we have a valid API server path then create the API server. For
now this has no functional change there is a default API server path in
the clap handling but rather prepares to do so optionally.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In case of the virtio frontend driver doesn't need interupts for
certain queue event, it may explicitly write VIRTIO_MSI_NO_VECTOR
to the virtio common configuration, or it may doesn't configure
the event type vector at all.
This patch initializes both MSI-X configuration vector and queue vector
with VIRTIO_MSI_NO_VECTOR, so that the backend drivers won't trigger
unexpected interrupts to the guest.
Signed-off-by: Zide Chen <zide.chen@intel.com>
Relying on a NVIDIA Tesla T4 card present in the SGX machine, this patch
enables baremetal VFIO testing, validated by running several NVIDIA
tools in the guest. The guest image has been prepared to include all the
software needed to run these tests.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit implements ARM PrimeCell General Purpose Input/Output
(GPIO) PL061 device specification.
Signed-off-by: Penny Zheng <penny.zheng@arm.com>
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This commit switches the default serial device from 16550 to the
Arm dedicated UART controller PL011. The `ttyAMA0` can be enabled.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Enabled all "ttyS0" related test cases:
- test_serial_off
- test_serial_tty
- test_serial_file
Enabled mandatory guest kernel driver for "ns16550a" on AArch64.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
On AArch64, interrupt controller (GIC) is emulated by KVM. VMM need to
set IRQ routing for devices, including legacy ones.
Before this commit, IRQ routing was only set for MSI. Legacy routing
entries of type KVM_IRQ_ROUTING_IRQCHIP were missing. That is way legacy
devices (like serial device ttyS0) does not work.
The setting of X86 IRQ routing entries are not impacted.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Extend the existing url_to_path() to take the URL string and then use
that to simplify the snapshot/restore code paths.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Now that virtio devices can be updated with add_memory_region(), there's
no need to keep update_memory() around.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Relies on the preliminary work allowing virtio devices to be updated
with a single memory at a time instead of updating the entire memory at
once.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Assuming vhost-user devices support CONFIGURE_MEM_SLOTS protocol
feature, we introduce a new method to the VirtioDevice trait in order to
update one single memory at a time.
In case CONFIGURE_MEM_SLOTS is not supported by the backend (feature not
acked), we fallback onto the current way of updating the memory
mappings, that is with SET_MEM_TABLE.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to support GET_MAX_MEM_SLOTS, ADD_MEM_REG and REM_MEM_REG, the
protocol feature CONFIGURE_MEM_SLOTS must be enabled.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to support GET_MAX_MEM_SLOTS, ADD_MEM_REG and REM_MEM_REG, the
protocol feature CONFIGURE_MEM_SLOTS must be enabled.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The latest vhost version adds the support for the new commands
get_max_mem_slots(), add_mem_region() and remove_mem_region(), all
related to the new vhost-user protocol feature CONFIGURE_MEM_SLOTS.
The vhost_user_backend crate is updated accordingly in order to support
these new commands, mostly related to the capability of updating the
guest memory mappings with a finer control than set_mem_table() command.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
On x86_64 architecture, multiple syscalls were missing when shutting
down the vhost-user-net device along with the VM. This was causing the
usual crash related to seccomp filters.
This commit adds these missing syscalls to fix the issue.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
There is no need to get the vring base when resetting the vhost-user
device. This was mostly ignored, but in some cases, it was causing some
actual errors.
A reset must simply be a combination of disabling the vrings along with
the reset of the owner.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Originally, VhostUserSetup is only used by vhost-user-fs. While
vhost-user-blk and vhost-user-net have their own error messages,
we rename VhostUserSetup to VhostUserFsSetup.
Signed-off-by: Jiachen Zhang <zhangjiachen.jaycee@bytedance.com>
The MMIO structure contains the length rather than the maximum address
so it is necessary to subtract the starting address from the end address
to calculate the length.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In the absence of a way of integration testing this testing that it
compiles is reasonable compromise.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Load the sections backed from the file into their required addresses in
memory and populate the HOB with details of the memory. Using the HOB
address initialize the TDX state in the vCPUs and finalize the TDX
configuration.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add API to the hypervisor interface and implement for KVM to allow the
special TDX KVM ioctls on the VM and vCPU FDs.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When booting with TDX no kernel is supplied as the TDFV is responsible
for loading the OS. The requirement to have the kernel is still
currently enforced at the validation entry point; this change merely
changes function prototypes and stored state to use Option<> to support.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add support extracting the sections out for a TDVF file which can be
then used to load the TDVF and TD HOB data into their appropriate
locations.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add the skeleton of the "tdx" feature with a module ready inside the
arch crate to store implementation details.
TEST=cargo build --features="tdx"
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
These need to be updated together as the kvm-ioctls depends upon a
strictly newer version of kvm-bindings which requires a rebase in the CH
fork.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When a vm is created with a pty device, on reboot the pty fd (sub
only) will only be associated with the vmm through the epoll event
loop. The fd being polled will have been closed due to the vm itself
dropping the pty files (and potentially reopening the fd index to a
different item making things quite confusing) and new pty fds will be
opened but not polled on for input.
This change creates a structure to encapsulate the information about
the pty fd (main File, sub File and the path to the sub File). On
reboot, a copy of the console and serial pty structs is then passed
down to the new Vm instance which will be used instead of creating a
new pty device.
This resolves the underlying issue from #2316.
Signed-off-by: William Douglas <william.r.douglas@gmail.com>
Now that virtio-mem devices can update VFIO mappings through dedicated
handlers, let's provide them from the DeviceManager.
Important to note these handlers should either be provided to virtio-mem
devices or to the unique virtio-iommu device. This must be mutually
exclusive.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Create two functions for registering/unregistering DMA mapping handlers,
each handler being associated with a VFIO device.
Whenever the plugged_size is modified (which means triggered by the
virtio-mem driver in the guest), the virtio-mem backend is responsible
for updating the DMA mappings related to every VFIO device through the
handler previously provided.
It's important to update the map when the handler is either registered
or unregistered as well, as we don't want to miss some plugged memory
that would have been added before the VFIO device is added to the VM.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Instead of letting the VfioPciDevice take the decision on how/when to
perform the DMA mapping/unmapping, we move this to the DeviceManager
instead.
The point is to let the DeviceManager choose which guest memory regions
should be mapped or not. In particular, we don't want the virtio-mem
region to be mapped/unmapped as it will be virtio-mem device
responsibility to do so.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
When memory is resized through ACPI, a new region is added to the guest
memory. This region must also be added to the corresponding memory zone
in order to keep everything in sync.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Replace "--monitor-fd" with "--event-monitor" which can either take
"fd=<int>" or "path=<path>" which can point to e.g. a named pipe and
allow more flexibility.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Since the SGX server is down for maintenance, all builds are waiting on
the node agent to answer, causing all PRs to be blocked.
Let's disable temporarily the SGX CI until the server is back up.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
So far we've only had the need to emulate one instruction. There is no
need to use a HashMap when a simple tuple for the initial mapping will
do.
We can bring back the HashMap once more sophisticated use cases surface.
No functional change.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
This avoids code complexity down the line when we get around
implementing Windows support.
No functional change.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Moving to the latest version of the rust-vmm/vhost crate, before it gets
published on crates.io.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In particular update for the vmm-sys-util upgrade and all the other
dependent packages. This requires an updated forked version of
kvm-bindings (due to updated vfio-ioctls) but allowed the removal of our
forked version of kvm-ioctls.
The changes to the API from kvm-ioctls and vmm-sys-util required some
other minor changes to the code.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The vhost crate from rust-vmm is ready, which is why we do the switch
from the Cloud Hypervisor fork to the upstream crate.
At the same time, we rename the crate from vhost_rs to vhost.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit moves both pci and vmm code from the internal vfio-ioctls
crate to the upstream one from the rust-vmm project.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that ExternalDmaMapping is defined in vm-device, let's use it from
there.
This commit also defines the function get_host_address_range() to move
away from the vfio-ioctls dependency.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By moving the trait and its VFIO implementation out of vfio-ioctls, we
give anticipate for the move to the vfio-ioctls from rust-vmm.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This has been lagging behind on an older Rust version and we have
enough coverage from our Jenkins CI agent.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This is only use in the integration test and was erroneously included in
the main binary dependencies.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The interrupt tests were not being run as they were erroneously under a
feature guard that does not exist in arch.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This removes the dependency on "tempdir" which in turn depends on the
large rand dependency chain.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The help information displayed for our `--disk` option is incorrect and
incomplete, e.g. missing the `direct` and `poll_queue` field.
Signed-off-by: Bo Chen <chen.bo@intel.com>
According to Intel's mnemonic (which is used by iced-x86) the first
argument is destination while the second is source.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
The main idea behind this commit is to remove all the complexity
associated with TX/RX handling for virtio-net. By using writev() and
readv() syscalls, we could get rid of intermediate buffers for both
queues.
The complexity regarding the TAP registration has been simplified as
well. The RX queue is only processed when some data are ready to be
read from TAP. The event related to the RX queue getting more
descriptors only serves the purpose to register the TAP file if it's not
already.
With all these simplifications, the code is more readable but more
performant as well. We can see an improvement of 10% for a single
queue device.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Currently script does not exit early if the image/firmware not present
in the host. We should not progress further if the images are not pre
downloaded.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
This crate exposes the abililty for the VMM to set a file that events
should be written to. The event!() macro provides an interface to report
those events allowing the specification of an event source, an event
type and optional extra data. This will be written to the provided file
descriptor as JSON data.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
We don't have an easy way to figure out if a GPA points to normal memory
or device memory, but the guest's normal memory regions shouldn't
overlap with device regions. We can simply try to do a normal memory
read / write, and proceed to do device memory read / write if that
fails.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
This function can then be used by the TDX code to allocate the memory at
specific locations required for the TDVF to run from.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
OVMF would use string IO on those ports. String IO has not been
implemented, so that leads to panics.
Skip them explicitly in MSHV. Leave a long-ish comment in code to
explain the situation. We should properly implement string IO once it
becomes feasible / necessary.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Update for clippy in Rust 1.50.0:
error: Unnecessary nested match
--> vmm/src/vm.rs:419:17
|
419 | / if let vm_device::BusError::MissingAddressRange = e {
420 | | warn!("Guest MMIO write to unregistered address 0x{:x}", gpa);
421 | | }
| |_________________^
|
= note: `-D clippy::collapsible-match` implied by `-D warnings`
help: The outer pattern can be modified to include the inner pattern.
--> vmm/src/vm.rs:418:17
|
418 | Err(e) => {
| ^ Replace this binding
419 | if let vm_device::BusError::MissingAddressRange = e {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ with this pattern
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#collapsible_match
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
If the function can never return an error this is now a clippy failure:
error: this function's return value is unnecessarily wrapped by `Result`
--> virtio-devices/src/watchdog.rs:215:5
|
215 | / fn set_state(&mut self, state: &WatchdogState) -> io::Result<()> {
216 | | self.common.avail_features = state.avail_features;
217 | | self.common.acked_features = state.acked_features;
218 | | // When restoring enable the watchdog if it was previously enabled. We reset the timer
... |
223 | | Ok(())
224 | | }
| |_____^
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#unnecessary_wraps
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This addresses a newly added clippy issue:
error: methods called `as_*` usually take self by reference or self by mutable reference; consider choosing a less ambiguous name
--> vm-migration/src/protocol.rs:57:34
|
57 | fn as_mut_bytes<T: Sized>(p: &mut T) -> &mut [u8] {
| ^^^^^^
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#wrong_self_convention
error: aborting due to 2 previous errors
error: could not compile `vm-migration`
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Depending on the host OS the code for looking up the time for the CMOS
make require extra syscalls to be permitted for the vCPU thread.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In case the VFIO device does not support MSI or MSI-X, the capabilities
should not be parsed, avoiding the exposure of unsupported capabilities.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Make sure to propagate the error coming from VfioDevice when trying to
enable INTx, MSI or MSI-X interrutps.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Since INTx is now supported for VFIO PCI devices, there is no more
limitations regarding VFIO, which means the documentation must be
updated accordingly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
With all the preliminary work done in the previous commits, we can
update the VFIO implementation to support INTx along with MSI and MSI-X.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Here we are adding the PCI routing table, commonly called _PRT, to the
ACPI DSDT. For simplification reasons, we chose not to implement PCI
links as this involves dynamic decision from the guest OS, which result
in lots of complexity both from an AML perspective and from a device
manager perspective.
That's why the _PRT creates a static list of 32 entries, each assigned
with the IRQ number previously reserved by the device manager.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to support INTx for PCI devices, each PCI device must be
assigned an IRQ. This is preliminary work to reserve 8 IRQs which will
be shared across the 32 PCI devices.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In anticipation for accessing the legacy interrupt manager from the
function creating a VFIO PCI device, we store it as part of the
DeviceManager, to make it available for all methods.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The DeviceManager already has a hold onto the MSI interrupt manager,
therefore there's no need to pass it through every function. Instead,
let's simplify the code by using the attribute from DeviceManager's
instance.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Both GIC and IOAPIC must implement a new method notifier() in order to
provide the caller with an EventFd corresponding to the IRQ it refers
to.
This is needed in anticipation for supporting INTx with VFIO PCI
devices.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In anticipation for supporting the notifier function for the legacy
interrupt source group, we need this function to return an EventFd
instead of a reference to this same EventFd.
The reason is we can't return a reference when there's an Arc<Mutex<>>
involved in the call chain.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Swap the last two parameters of guest_mem_{read,write} to be consistent
with other read / write functions.
Use more descriptive parameter names.
No functional change.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
This reflects that it generates CPUID state used across all vCPUs.
Further ensure that errors from this function get correctly propagated.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Move the code for populating the CPUID with KVM HyperV emulation details from
the per-vCPU CPUID handling code to the shared CPUID handling code.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Move the code for populating the CPUID with details of the CPU
identification from the per-vCPU CPUID handling code to the shared CPUID
handling code.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Move the code for populating the CPUID with details of the maximum
address space from the per-vCPU CPUID handling code to the shared CPUID
handling code.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add the ability for cloud-hypervisor to create, manage and monitor a
pty for serial and/or console I/O from a user. The reasoning for
having cloud-hypervisor create the ptys is so that clients, libvirt
for example, could exit and later re-open the pty without causing I/O
issues. If the clients were responsible for creating the pty, when
they exit the main pty fd would close and cause cloud-hypervisor to
get I/O errors on writes.
Ideally the main and subordinate pty fds would be kept in the main
vmm's Vm structure. However, because the device manager owns parsing
the configuration for the serial and console devices, the information
is instead stored in new fields under the DeviceManager structure
directly.
From there hooking up the main fd is intended to look as close to
handling stdin and stdout on the tty as possible (there is some future
work ahead for perhaps moving support for the pty into the
vmm_sys_utils crate).
The main fd is used for reading user input and writing to output of
the Vm device. The subordinate fd is used to setup raw mode and it is
kept open in order to avoid I/O errors when clients open and close the
pty device.
The ability to handle multiple inputs as part of this change is
intentional. The current code allows serial and console ptys to be
created and both be used as input. There was an implementation gap
though with the queue_input_bytes needing to be modified so the pty
handlers for serial and console could access the methods on the serial
and console structures directly. Without this change only a single
input source could be processed as the console would switch based on
its input type (this is still valid for tty and isn't otherwise
modified).
Signed-off-by: William Douglas <william.r.douglas@gmail.com>
This thread is virtio-net specific, so it is not handled in the common
virtio device code.
The non-vhost implementation resumes the thread itself. Do the same
thing for vhost-user-net.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Use the newly added hugepages_size option if provided by the user to
pick a huge page size when creating the memfd region. If none is
specified use the system default.
Sadly different huge pages cannot be tested by an integration test as
creating a pool of the non-default size cannot be done at runtime
(requires kernel to be booted with certain parameters.)
TETS=Manually tested with a kernel booted with both 1GiB and 2MiB huge
pages (hugepagesz=1G hugepages=1 hugepagesz=2M hugepages=512)
Fixes: #2230
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Including a warning that the user is respsonsible for ensuring that they
have sufficient pages of the specified size.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This allows the user to use an alternative huge page size otherwise the
default size will be used.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This commit introduces a new information to the VirtioMemZone structure
in order to know if the memory zone is backed by hugepages.
Based on this new information, the virtio-mem device is now able to
determine if madvise(MADV_DONTNEED) should be performed or not. The
madvise documentation specifies that MADV_DONTNEED advice will fail if
the memory range has been allocated with some hugepages.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Signed-off-by: Hui Zhu <teawater@antfin.com>
This commit performs some refactoring to make all functions a method
from a specific object, and in particular methods for MemEpollHandler.
The point is to simplify the code to make it more readable.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Adjust the code to comply better with the virtio-mem specification by
adding some validation for the virtio-mem configuration, but also by
updating the virtio-mem configuration itself.
Nowhere in the virtio-mem specification is stated the usable region size
must be adjusted everytime the plugged size changes. For simplification
reason, and without going against the specification, the usable region
size is now kept static, setting its value to the size of the whole
region.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By introducing a ResizeSender object, we avoid having a Resize clone
with a different content than the original Resize object.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
MOV R/RM is a special case of MOVZX, so we generalize the mov_r_rm macro
to make it support both instructions.
Fixes: #2227
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The vhd module is the implementation of the VHD specification, which is
why it is important to unit test it.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Let's create a fixed VHD disk file from the existing RAW file thanks to
qemu-img, and create a new integration test to validate that
Cloud-Hypervisor can boot VHD disk image.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Relying on the simplified version of the synchronous support for RAW
disk files, the new fixed_vhd_sync module in the block_util crate
introduces the synchronous support for fixed VHD disk files.
With this patch, the fixed VHD support is complete as it is implemented
in both synchronous and asynchronous versions.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Using directly preadv and pwritev, we can simply use a RawFd instead of
a file, and we don't need to use the more complex implementation from
the qcow crate.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit adds the asynchronous support for fixed VHD disk files.
It introduces FixedVhd as a new ImageType, moving the image type
detection to the block_util crate (instead of qcow crate).
It creates a new vhd module in the block_util crate in order to handle
VHD footer, following the VHD specification.
It creates a new fixed_vhd_async module in the block_util crate to
implement the asynchronous version of fixed VHD disk file. It relies on
io_uring.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
_EJx built in should not return.
dsdt.dsl 813: Return (CEJ0 (0x00))
Warning 3104 - ^ Reserved method should not return a value (_EJ0)
dsdt.dsl 813: Return (CEJ0 (0x00))
Error 6080 - ^ Called method returns no value
Fixes: #2216
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The mutex timeout should be 0xffff rather than 0xfff to disable the
timeout feature.
dsdt.dsl 745: Acquire (\_SB.PRES.CPLK, 0x0FFF)
Warning 3130 - ^ Result is not used, possible operator timeout will be missed
dsdt.dsl 767: Acquire (\_SB.PRES.CPLK, 0x0FFF)
Warning 3130 - ^ Result is not used, possible operator timeout will be missed
dsdt.dsl 775: Acquire (\_SB.PRES.CPLK, 0x0FFF)
Warning 3130 - ^ Result is not used, possible operator timeout will be missed
Fixes: #2216
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
By using `net_util::open_tap` to create the TAP interface, the created
interface will be deleted when the returned variable (`net_utils::Tap`)
is dropped.
Signed-off-by: Bo Chen <chen.bo@intel.com>
This patch enables multi-queue support for creating virtio-net devices by
accepting multiple TAP fds, e.g. '--net fds=3:7'.
Fixes: #2164
Signed-off-by: Bo Chen <chen.bo@intel.com>
This helper can open a TAP device and configure the interface on it. If
the device needs to be opened multiple times for MQ then it also handles
that correctly.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Because of the behavior of the NVIDIA proprietary driver, we can't
expect NVIDIA cards with only MSI support to be functioning correctly
after they've been passed through with Cloud-Hypervisor.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Building with 1.51 nightly produces the following warning:
warning: unnecessary trailing semicolon
--> vmm/src/device_manager.rs:396:6
|
396 | };
| ^ help: remove this semicolon
|
= note: `#[warn(redundant_semicolons)]` on by default
warning: 1 warning emitted
Signed-off-by: Wei Liu <liuwe@microsoft.com>
The unit tests ask the Linux kernel to generate a TAP device name on
demand by passing in a format string. I suspect, but haven't been able
to confirm that there might be a rare race that triggers when creating
lots of devices in a short period of time. This is appearing in our unit
test as the occassional flake of the test_tap_read() which although it
has successfully created the device it fails to set the IP address on it
when looking it back up by it's name.
Since this is the most frequent cause of failures on our CI use a lock
to ensure that multiple TAP devices are not created simultaneously.
Fixes: #2135
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This skeleton commit brings in the support for compiling aarch64 with
the "acpi" feature ready to the ACPI enabling. It builds on the work to
move the ACPI hotplug devices from I/O ports to MMIO and conditionalises
any code that is x86_64 only (i.e. because it uses an I/O port.)
Filling in the aarch64 specific details in tables such as the MADT it
out of the scope.
See: #2178
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
e.g.
scripts/dev_cli.sh tests --integration -- --test-filter test_watchdog
This used to be supported by passing "$@" but was broken when multiple
hypervisor support was added.
Fixes: #2182
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Since QCOW and RAW synchronous implementation are very close, it makes
sense to introduce some common functions that can be shared between
these two.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
It might be useful debugging information for the user to know what kind
of disk file implementation is in use.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that BlockIoUring is the only implementation of virtio-block,
handling both synchronous and asynchronous backends based on the
AsyncIo trait, we can rename it to Block.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that both synchronous and asynchronous backends rely on the
asynchronous version of virtio-block (namely BlockIoUring), we can
get rid of the synchronous version (namely Block).
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the synchronous QCOW file implementation present in the qcow
crate, we created a new qcow_sync module in block_util that ports this
synchronous implementation to the AsyncIo trait.
The point is to reuse virtio-blk asynchronous implementation for both
synchronous and asynchronous backends.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the synchronous RAW file implementation present in the qcow
crate, we created a new raw_sync module in block_util that ports this
synchronous implementation to the AsyncIo trait.
The point is to reuse virtio-blk asynchronous implementation for both
synchronous and asynchronous backends.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the new DiskFile and AsyncIo traits, the implementation of
asynchronous block support does not have to be tied to io_uring anymore.
Instead, the only thing the virtio-blk implementation knows is that it
is using an asynchronous implementation of the underlying disk file.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Both DiskFile and AsyncIo traits are introduced to allow all kind of
files (RAW, QCOW, VHD) to be able to handle asynchronous access to the
underlying file.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Currently the GED control is in a fixed I/O port address but instead use
an MMIO address that has been chosen by the allocator.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This patch refines the sccomp filter list for the vCPU thread, as we are
no longer spawning virtio-device threads from the vCPU thread.
Fixes: #2170
Signed-off-by: Bo Chen <chen.bo@intel.com>
Even though the driver can provide fewer queues than those advertised
for some device types their is a minimum number that is required for
operation.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
It is permissable for the driver to program fewer queues than offered by
the device. Filter the queues so that only the ready ones are included
and check that they have valid addresses configured.
Fixes: #2136
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Don't assume that the number of queues provided match the number of
queues offered. The virtio spec allows the driver to program fewer
queues.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
We should use full memory barrier to ensure both guest and us
can see the correct avail_idx and avail_event_idx. Something
like this pattern:
VM: CLH:
update vring.avail->idx update avail_event_idx
mb() mb()
read avail_event_idx read vring.avail->idx
Signed-off-by: Xie Yongji <xieyongji@bytedance.com>
Rather than having to give and return the ioeventfd used for a device
clone them each time. This will make it simpler when we start handling
the driver enabling fewer queues than advertised by the device.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
We have killed the device thread by writing to the exit EventFd but we
should wait for them to quit to ensure consistency.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This will lead to the triggering of an ACPI button inside the guest in
order to cleanly shutdown the guest.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Use the ACPI GED device to trigger a notitifcation of type
POWER_BUTTON_CHANGED which will ultimately lead to the guest being
notified.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
If a GED event of type 0x8 (for power button) is received notify the
guest OS that this button device has been activated.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Renamed this bitfield as it will also be used for non-hotplug purposes
such as synthesising a power button.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In order to make the thread naming more useful derive their name from
the device id (which can be supplied by the user) and a device specific
suffix that has details of the individual queue (or queue pair.)
e.g.
rob@artemis:~$ pstree -p -c -l -t `pidof cloud-hypervisor`
cloud-hyperviso(27501)─┬─{_console}(27525)
├─{_disk0_q0}(27529)
├─{_disk0_q1}(27532)
├─{_net1_ctrl}(27533)
├─{_net1_qp0}(27534)
├─{_net1_qp1}(27535)
├─{_rng}(27526)
├─{http-server}(27504)
├─{seccomp_signal_}(27502)
├─{signal_handler}(27523)
├─{vcpu0}(27520)
├─{vcpu1}(27522)
└─{vmm}(27503)
Fixes: #2077
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The Windows image is quite large (about 20GiB), hence it takes some time
to copy it for every test in order to avoid potential corruption.
One way to mitigate that without compromising on safety between each
test is by using device mapper. By creating a read-only base, we ensure
the image won't be modified by any of the tests, and by creating one
snapshot for each test, we avoid copying the entire image each time.
A dedicated Copy On Write disk image is created to handle any change
that might be performed on the base image, letting the tests behave as
expected.
Fixes#2155
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By relying on the Guest object, Windows dedicated tests copy the Windows
guest image before booting from it. The point being to avoid corruption
between multiple tests. This is already how the rest of the integration
tests work, Windows tests were the only ones missing this feature.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The tight coupling of these packages caused issues for dependabot trying
to update them separately so instead combine them together into one
update.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This image does not have the pollinate service which can sometimes fail
and prevent SSH from starting as it marks itself as a prerequisite. This
service will never fully succeed as it tries to make a network
connection which will fail inside our test VMs.
Fixes: #2113
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This allows the removal of most of the custom patches in particular all
the virtio-fs ones have been integrated.
Fixes: #2070
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The ch-5.10.6 branch is rebased on top of Linux stable and includes the
latest virtio-iommu support patches.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Using --net=host is not necessary for any of the integration tests, so
let's use the default network option called "bridge".
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Older libc (like RHEL7) uses open() rather than openat(). This was
demonstrated through a failure to open /etc/localtime as used by
gmtime() libc call trigged from the vCPU thread (CMOS device.)
Fixes: #2111
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Because we're back on transient builder, let's download the image
everytime.
This reverts commit b5653d5278.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Some sporadic failures were due to an early connection to the VM while
it was not fully ready. Increasing sleep times fixes these issues.
Fixes#2104
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Sometimes when running under the CI tests fail due to a barrier not
being released and the guest blocks on an MMIO write. Add further
debugging to try and identify the issue.
See: #2118
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Given we already check the connected IP address matches the expected
guest IP address, the check on the "booted" message is not needed.
Fixes: #2117
Signed-off-by: Bo Chen <chen.bo@intel.com>
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
On aarch64, the openat() syscall was missing from the seccomp filters
list, preventing the test_watchdog from running properly.
Fixes#2103
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Temporarily disable fast failing to try and make progress on CI
stabilisation to allow more test coverage.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This test is very flaky and regularly causing CI failures. Until we can
identify the root cause we should disable this test.
See: #2103
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Simplify our image handling by not copying both QCOW2 and raw images for
every test. Allow the test to choose QCOW2 or raw by specifying the
image name manually. A follow on patch will add explicity QCOW2 tests.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When an SSH command fails we want to be able to see, via a panic() why
and where it failed. Replace use of .unwrap_or_default() from SSH
command calls to ensure that we can see the location of the panic.
Also enhance the existing SSH output code to show the error if there is
one.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Based on the LKML feedback, the devices under /dev/sgx/* are
not justified. SGX RFC v40 moves the SGX device nodes to /dev/sgx_*
and this is reflected in kvm-sgx (next branch) too.
Update cloud-hypervisor code and documentation to follow this.
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
This reverts commit 7f1f335950.
This breaks the compilation on Fedora/RedHat machines by requiring
dpkg-deb in the build.rs
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This allows the removal of most of the custom patches in particular all
the virtio-fs ones have been integrated.
Partially fixes: #2070
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
If the vCPU thread calls log!() the time difference between the call
time and the boot up time is reported. On most environments and
architectures this covered by a vDSO call rather than a syscall. However
on some platforms this turns into a syscall.
Fixes: #2080
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
error: field assignment outside of initializer for an instance created with Default::default()
Error: --> hypervisor/src/kvm/mod.rs:1239:9
|
1239 | state.mp_state = self.get_mp_state()?;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= note: `-D clippy::field-reassign-with-default` implied by `-D warnings`
note: consider initializing the variable with `kvm::aarch64::VcpuKvmState { mp_state: self.get_mp_state()?, ..Default::default() }` and removing relevant reassignments
--> hypervisor/src/kvm/mod.rs:1237:9
|
1237 | let mut state = CpuState::default();
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#field_reassign_with_default
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
error: field assignment outside of initializer for an instance created with Default::default()
--> virtio-devices/src/mem.rs:496:9
|
496 | resp.resp_type = resp_type;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
note: consider initializing the variable with `mem::VirtioMemResp { resp_type: resp_type, ..Default::default() }` and removing relevant reassignments
--> virtio-devices/src/mem.rs:495:9
|
495 | let mut resp = VirtioMemResp::default();
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#field_reassign_with_default
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
We must explicitly mark these values as u8 as the function that consumes
them takes a T and needs to use the specific width.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
error: field assignment outside of initializer for an instance created with Default::default()
--> hypervisor/src/kvm/mod.rs:318:9
|
318 | cap.cap = KVM_CAP_SPLIT_IRQCHIP;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= note: `-D clippy::field-reassign-with-default` implied by `-D warnings`
note: consider initializing the variable with `kvm_bindings::kvm_enable_cap { cap: KVM_CAP_SPLIT_IRQCHIP, ..Default::default() }` and removing relevant reassignments
--> hypervisor/src/kvm/mod.rs:317:9
|
317 | let mut cap: kvm_enable_cap = Default::default();
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#field_reassign_with_default
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
error: this returns a `Result<_, ()>
--> net_util/src/mac.rs:68:5
|
68 | pub fn from_bytes(src: &[u8]) -> Result<MacAddr, ()> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= note: `-D clippy::result-unit-err` implied by `-D warnings`
= help: use a custom Error type instead
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#result_unit_err
Replace with std::io::Error like other locations in the same file.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
With Rust 1.49 using attributes on a function parameter is not allowed.
The recommended workaround is to put it in a new block.
error[E0658]: attributes on expressions are experimental
--> vmm/src/memory_manager.rs:698:17
|
698 | #[cfg(target_arch = "x86_64")]
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= note: see issue #15701 <https://github.com/rust-lang/rust/issues/15701> for more information
error: removing an expression is not supported in this position
--> vmm/src/memory_manager.rs:698:17
|
698 | #[cfg(target_arch = "x86_64")]
|
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The boot time for direct kernel boot based tests is significantly
quicker than booting via the firmware and stock kernel as it triggers a
reboot during the boot process due to the initrd handling.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When doing a direct kernel boot only have console=ttyS0 in the command
line if we are explicitly testing the serial output. The default
behaviour is `--serial null` so this output will not be visible but will
trigger a KVM exit for every byte which is very costly when running
under nested virtualization.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
On the CI we are seeing issues with the activation barriers not being released:
cloud-hypervisor: 12.452434193s: INFO:vmm/src/vm.rs:413 -- Waiting for barrier
cloud-hypervisor: 12.452499794s: INFO:virtio-devices/src/block.rs:382 -- Changing cache mode to writeback
cloud-hypervisor: 12.452605195s: INFO:vmm/src/vm.rs:413 -- Waiting for barrier
cloud-hypervisor: 12.452684596s: INFO:virtio-devices/src/transport/pci_device.rs:671 -- Waiting for barrier
cloud-hypervisor: 12.452708196s: INFO:virtio-devices/src/transport/pci_device.rs:673 -- Barrier released
cloud-hypervisor: 12.452717596s: INFO:vmm/src/vm.rs:415 -- Barrier released
Add some debugging to try and identify the vause of this issue.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Now that we can support supplying an fd to the TAP device we can support
the MACVTAP and MACVLAN use cases.
Fixes: #872
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add an "fd=" parameter to allow specifying a TAP fd to use. Currently
only one fd for one queue pair is supported.
Fixes: #2052
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add support for creating virtio-net device from existing TAP fd.
Currently only a single fd and thus no-more than 2 queues (one pair) is
suppored.
Fixes: #2052
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Ultimately this will allow the creation of a virtio-net device that is
backed by a file descriptor.
This function ensures that the TAP device is correctly setup with
offloading and non-blocking.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
And fix related warnings: op_kind and op_register are being deprecated
as they might panic.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Starting the virtio device threads from the VMM thread has slowed down
the start of the VM when running on a highly contested system like the
CI.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When a device is ready to be activated signal to the VMM thread via an
EventFd that there is a device to be activated. When the VMM receives a
notification on the EventFd that there is a device to be activated
notify the device manager to attempt to activate any devices that have
not been activated.
As a side effect the VMM thread will create the virtio device threads.
Fixes: #1863
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
We need to be able to return the barrier from the code that prepares to
activate the virtio device. This triggered by a write to the
configuration fields stored in the PCI BAR. Since bars can be accessed
by both memory mapping and through PCI config I/O several prototypes
must be changed.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This can be uses to indicate to the caller that it should wait on the
barrier before returning as there is some asynchronous activity
triggered by the write which requires the KVM exit to block until it's
completed.
This is useful for having vCPU thread wait for the VMM thread to proceed
to activate the virtio devices.
See #1863
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Currently these two macros(msr, msr_data) reside both on kvm and mshv
module. Definition is same for both module. Moving them to arch/x86
module eliminates redundancy and makes more sense.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
On the CI we are seeing that sometimes the epoll is receiving these
errors which do not indicate a failure but that we should retry.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Since we can't test mshv and kvm at the same time, --all-features no
longer work.
We factorize all, non-hypervisor related features into a common set and
mix that with either mshv and kvm.
Co-Developed-by: Samuel Ortiz <sameo@linux.intel.com>
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Co-Developed-by: Wei Liu <liuwe@microsoft.com>
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Signed-off-by: Muminul Islam <muislam@microsoft.com>
We don't have IrqFd and IOEventFd support in the kernel for now.
So an emulation layer is needed. In the future, we will be adding this
support in the kernel.
Co-Developed-by: Wei Liu <liuwe@microsoft.com>
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Signed-off-by: Muminul Islam <muislam@microsoft.com>
vmmops trait object is needed to get access some
of the upper level vmm functionalities i.e guest
memory access, IO read write etc.
Co-Developed-by: Praveen Paladugu <prapal@microsoft.com>
Signed-off-by: Praveen Paladugu <prapal@microsoft.com>
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Adding hv_state (hyperv state) to Vm and Vcpu struct for mshv.
This state is needed to keep some kernel data(for now hypercall page)
in the vmm.
Co-Developed-by: Praveen Paladugu <prapal@microsoft.com>
Signed-off-by: Praveen Paladugu <prapal@microsoft.com>
Signed-off-by: Muminul Islam <muislam@microsoft.com>
This is the initial folder structure of the mshv module inside
the hypervisor crate. The aim of this module is to support Microsoft
Hyper-V as a supported Hypervisor.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
There are some code base and function which are purely KVM specific for
now and we don't have those supports in mshv at the moment but we have plan
for the future. We are doing a feature guard with KVM. For example, KVM has
mp_state, cpu clock support, which we don't have for mshv. In order to build
those code we are making the code base for KVM specific compilation.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
The customized hashmap macro can't be lifted to common MockVMM code.
MockVMM only needs a collection to iterate over to get initial register
states. A vector is just as good as a hashmap.
Switch to use a vector to store initial register states. This allows us
to drop the hashmap macro everywhere.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Unfortunately Rust stable does not yet have inline ASM support the flag
calculation will have to be implemented in software.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
nightly / beta are not required actions but if they fail then all other
builds will be cancelled and so prevent the PR from being cleared as
buildable.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Unfortunately it seems patch entries are ignored when obtaining
dependencies from another workspace.
Remove the problematic kvm-ioctls and kvm-bindings patch entries and use
the forked repository directly.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When using an PIO write to 0x80 which is a special case handle that and
then return without going through the resolve.
This removes an extra warning that is reported.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
There is no need to have a pair of curly brackets for structures without
any member.
No functional change.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
The mapping between code and its handler is static. We can drop the
HashMap in favour of a static match expression.
This has two benefits:
1. No more memory allocation and deallocation for the HashMap.
2. Shorter look-up time.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
When a total ordering between multiple atomic variables is not required
then use Ordering::Acquire with atomic loads and Ordering::Release with
atomic stores.
This will improve performance as this does not require a memory fence
on x86_64 which Ordering::SeqCst will use.
Add a comment to the code in the vCPU handling code where it operates on
multiple atomics to explain why Ordering::SeqCst is required.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The DeviceNode cannot be fully represented as it embeds a Rust style
enum (i.e. with data) which is instead represented by a simple
associative array.
Fixes: #1167
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When the x86 instruction decoder tells us about some missing bytes from
the instruction stream, we call into the platform fetch method and
emulate one last instruction.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
In preparation for the instruction fetching step, we modify the decoding
loop so that we can check what the last decoding error is.
We also switch to explictly using decode_out() which removes a 32 bytes
copy compared to decode().
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
In order to validate emulated memory accesses, we need to be able to get
all the segments descriptor attributes.
This is done by abstracting the SegmentRegister attributes through a
trait that each hypervisor will have to implement.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
We need to be able to build the CPU mode from its state in order to
start implementing mode related checks in the x86 emulator.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
If we rely on timeouts at the top level we can get builds being aborted
simply because they took too long to be scheduled rather than because
the actual integration tests took too long.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The MockVMM platform will be used by other instructions emulation
implementations, but also by the emulator framework.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The observation here is PlatformEmulator can be seen as the context for
emulation to take place. It should be rather easy to construct a context
that satisfies the lifetime constraints for instruction emulation.
The thread doing the emulation will have full ownership over the
context, so this removes the need to wrap PlatformEmulator in Arc and
Mutex, as well as the need for the context to be either Clone or Copy.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
The emulator gets a CPU state from a CpuStateManager instance, emulates
the passed instructions stream and returns the modified CPU state.
The emulator is a skeleton for now since it comes with an empty
instruction mnemonic map.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
And an InstructionMap helper structure to map x86 mnemonic codes
to instruction handlers.
Any instruction emulation implementation should then boil down with
implementing InstructionHandler for any supported mnemonic.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Minimal will be defined by the amount of emulated instructions.
Carrying all GPRs, all CRs, segment registers and table registers should
cover quite a few instructions.
Co-developed-by: Wei Liu <liuwe@microsoft.com>
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
For efficiently emulating x86 instructions, we need to build and pass a
CPU state copy/reference to instruction emulation handlers. Those handlers
will typically modify the CPU state and let the caller commit those
changes back through the PlatformEmulator trait set_cpu_state method.
Hypervisors typically have internal CPU state structures, that maps back
to the correspinding kernel APIs. By implementing the CpuState trait,
instruction emulators will be able to directly work on CPU state
instances that are directly consumable by the underlying hypervisor and
its kernel APIs.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
In order to emulate instructions, we need a way to get access to some of
the guest resources. The PlatformEmulator interface provides guest
memory and CPU state access to emulator implementations.
Typically, an hypervisor will implement PlatformEmulator for architecture
specific instruction emulators to build their framework on top of.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
We will need the GDT API for the hypervisor's x86 instruction
emulator implementation, it's better if the arch crate depends on the
hypervisor one rather than the other way around.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The configuration is stored separately to the Vm in the VMM. The failure
to store the config was preventing the VM from shutting down correctly
as Vmm::vm_delete() checks for the presence of the config.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Rather return the None to the caller to handle instead. This removes the
source of a potential panic.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The live migration support added use of this ioctl but it wasn't
included in the permitted list.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This interface is used by the vCPU thread to delegate responsibility for
handling MMIO/PIO operations and to support different approaches than a
VM exit.
During profiling I found that we were spending 13.75% of the boot CPU
uage acquiring access to the object holding the VmmOps via
ArcSwap::load_full()
13.75% 6.02% vcpu0 cloud-hypervisor [.] arc_swap::ArcSwapAny<T,S>::load_full
|
---arc_swap::ArcSwapAny<T,S>::load_full
|
--13.43%--<hypervisor::kvm::KvmVcpu as hypervisor::cpu::Vcpu>::run
std::sys_common::backtrace::__rust_begin_short_backtrace
core::ops::function::FnOnce::call_once{{vtable-shim}}
std::sys::unix::thread::Thread::new::thread_start
However since the object implementing VmmOps does not need to be mutable
and it is only used from the vCPU side we can change the ownership to
being a simple Arc<> that is passed in when calling create_vcpu().
This completely removes the above CPU usage from subsequent profiles.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
As we switched to focal for this test we no longer get any output during
the boot unless serial is used over virtio-console.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
There have been a lot of flakes around tests such as
test_virtio_fs_hotplug_dax_on_w_vhost_user_fs_daemon() or
test_virtio_fs_hotplug_dax_on() which all try and hotplug memory.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add config parameter to --disk called "_disable_io_uring" (the
underscore prefix indicating it is not for public consumpion.) Use this
option to disable io_uring if it would otherwise be used.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
With the removal of vhost-user self-spawning support we should migrate
the tests to use the binaries so that we can remove the functionality
from the cloud-hypervisor binary itself.
See: #1925
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Now the VM is paused/resumed by the migration process itself.
0. The guest configuration is sent to the destination
1. Dirty page log tracking is started by start_memory_dirty_log()
2. All guest memory is sent to the destination
3. Up to 5 attempts are made to send the dirty guest memory to the
destination...
4. ...before the VM is paused
5. One last set of dirty pages is sent to the destination
6. The guest is snapshotted and sent to the destination
7. When the migration is completed the destination unpauses the received
VM.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This allows code running in the VMM to access the VM's MemoryManager's
functionality for managing the dirty log including resetting it but also
generating a table.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Prior to sending the memory the full state is not needed only the
configuration. This is sufficient to create the appropriate structures
in the guest and have the memory allocations ready for filling.
Update the protocol documentation to add a separate config step and move
the state to after the memory is transferred. As the VM is created in a
separate step to restoring it the requires a slightly different
constructor as well as saving the VM object for the subsequent commands.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In order to do this we must extend the MemoryManager API to add the
ability to specify the tracking of the dirty pages when creating the
userspace mappings and also keep track of the userspace mappings that
have been created for RAM regions.
Currently the dirty pages are collected into ranges based on a block
level of 64 pages. The algorithm could be tweaked to create smaller
ranges but for now if any page in the block of 64 is dirty the whole
block is added to the range.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
While the addressable space size reduction of 4k in necessary due to
the Linux bug, the 64k alignment of the addressable space size is
required by Windows. This patch satisfies both.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
Cloning the ArcSwapOption (like the ArcSwap) does not act like a
.clone() on an Arc, instead an entirely new ArcSwap is created with the
same contents. To correctly share the ArcSwap needs to be placed inside
an Arc.
See: 2433d5719b (diff-6c6d94533c44c19bd1416ef17bad1a878e63dca6e98d59181228fbe8f967c62bR6)
Due to this being wrongly used ::clone() was removed from
ArcSwap/ArcSwapOption in 1.0.0.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This is tested by:
Source VMM:
target/debug/cloud-hypervisor --kernel ~/src/linux/vmlinux \
--pmem file=~/workloads/focal.raw --cpus boot=1 \
--memory size=2048M \
--cmdline"root=/dev/pmem0p1 console=ttyS0" --serial tty --console off \
--api-socket=/tmp/api1 -v
Destination VMM:
target/debug/cloud-hypervisor --api-socket=/tmp/api2 -v
And the following commands:
target/debug/ch-remote --api-socket=/tmp/api1 pause
target/debug/ch-remote --api-socket=/tmp/api2 receive-migration unix:/tmp/foo &
target/debug/ch-remote --api-socket=/tmp/api1 send-migration unix:/tmp/foo
target/debug/ch-remote --api-socket=/tmp/api2 resume
The VM is then responsive on the destination VMM.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This allows the code to be reused when creating the VM from a snapshot
when doing VM migration.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add API entry points with stub implementation for sending and receiving
a VM from one VMM to another.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Due to a known limitation in OpenAPITools/openapi-generator tool,
it's impossible to send go zero types, like false and 0 to
cloud-hypervisor because `omitempty` is added if a field is not
required.
Set cache_size, dax, num_queues and queue_size as required to remove
`omitempty` from the json tag.
fixes#1961
Signed-off-by: Julio Montes <julio.montes@intel.com>
This also removes the need to lookup up the "exe" symlink for finding
the VMM executable path.
Fixes: #1925
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The integration script failed to obtain the new guest kernel commit when
the code folder had been existing on CI machine.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
The logic to handle AArch64 system event was: SHUTDOWN and RESET were
all treated as RESET.
Now we handle them differently:
- RESET event will trigger Vmm::vm_reboot(),
- SHUTDOWN event will trigger Vmm::vm_shutdown().
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Now Vcpu::run() returns a boolean value to VcpuManager, indicating
whether the VM is going to reboot (false) or just continue (true).
Moving the handling of hypervisor VCPU run result from Vcpu to
VcpuManager gives us the flexibility to handle more scenarios like
shutting down on AArch64.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Set the test case test_snapshot_restore X86 only, instead of excluding
it from test command line.
The command line option was added because we used to support migration
with Virtio-MMIO, but not Virtio-PCI.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Rather than filling the guest memory from a file at the point of the the
guest memory region being created instead fill from the file later. This
simplifies the region creation code but also adds flexibility for
sourcing the guest memory from a source other than an on disk file.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Tests not ported include 1) the ones that start guest VMs without
network (e.g. test_net_hotplug, test_initramfs), 2) test_vfio that
involves l2 guest. Also, some tests that use bionic guest image are
given extended timeout (120s) for 'wait_vm_boot'.
Signed-off-by: Bo Chen <chen.bo@intel.com>
Instead of waiting blindly with fixed amount of sleeping time, we can
use the `wait-timeout` crate to explicitly wait VM shutdown (with a
timeout). It can reduces the execution time of some tests
substantially. Also, this patch increases the `shutdown` timeout for
'test_reboot', which should fix the recent sporadic failures on this
test.
Signed-off-by: Bo Chen <chen.bo@intel.com>
Address build failure from activity in the development virtio-fs branch
by using the stable fork.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Instead of blindly waiting for 20-40s for the guest VM to boot, this
patch waits the notification from the guest VM explicitly by using a
simple TcpListener on the host and a custom systemd service in the
guest.
This patch also ported few tests to use this new machanism, while more
tests are to be ported.
Signed-off-by: Bo Chen <chen.bo@intel.com>
As a mirror of bdbea19e23 which ensured
that GuestMemoryMmap::read_exact_from() was used to read all the file to
the region ensure that all the guest memory region is written to disk.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Use a Result<> type with an error to simplify the code in start_vmm().
This will also make it easier to add cleanup funtionality.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This gives a nicer user experience and this error can now be used as the
source for other errors based off this.
See: #1910
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Now that we that our CI is running with a kernel that is new enough to
support io_uring we can turn this feature on by default.
See: #1561
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Any occurrence of of a variable containing `ext_region` is replaced with
the less confusing name `saved_region`. The point is to clearly identify
the memory regions that might have been saved during a snapshot, while
the `ext` standing for `external` was pretty unclear.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In the context of saving the memory regions content through snapshot,
using the term "backing file" brings confusion with the actual backing
file that might back the memory mapping.
To avoid such conflicting naming, the 'backing_file' field from the
MemoryRegion structure gets replaced with 'content', as this is
designating the potential file containing the memory region data.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Use GuestRegionMmap::read_exact_from() to ensure that all of the file is
read into the guest. This addresses an issue where
GuestRegionMmap::read_from() was only copying the first 2GiB of the
memory and so lead to snapshot-restore was failing when the guest RAM
was 2GiB or greater.
This change also propagates any error from the copying upwards.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Split out the HTTP request handling code from ch-remote into a new
crate which can be used in other places where talking to the API server
by HTTP is necessary.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Use the trait for Read/Write rather than specifying the concrete type.
This allows for the functionality to be used for different socket types.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When restoring if a region of RAM is backed by anonymous memory i.e from
memfd_create() then copy the contents of the ram from the file that has
been saved to disk.
Previously the code would map the memory from that file into the guest
using a MAP_PRIVATE mapping. This has the effect of
minimising the restore time but provides an issue where the restored VM
does not have the same structure as the snapshotted VM, in particular
memory is backed by files in the restored VM that were anonymously
backed in the original.
This creates two problems:
* The snapshot data is mapped from files for the pages of the guest
which prevents the storage from being reclaimed.
* When snapshotting again the guest memory will not be correctly saved
as it will have looked like it was backed by a file so it will not be
written to disk but as it is a MAP_PRIVATE mapping the changes will
never be written to the disk again. This results in incorrect
behaviour.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Now that virtio-balloon is not declared as part of the --memory
parameter, the integration tests are updated to keep the correct
behavior.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The standalone `--balloon` parameter being fully functional at this
point, we can get rid of the balloon options from the --memory
parameter.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that we have a new dedicated way of asking for a balloon through the
CLI and the REST API, we can move all the balloon code to the device
manager. This allows us to simplify the memory manager, which is already
quite complex.
It also simplifies the behavior of the balloon resizing command. Instead
of providing the expected size for the RAM, which is complex when memory
zones are involved, it now expects the balloon size. This is a much more
straightforward behavior as it really resizes the balloon to the desired
size. Additionally to the simplication, the benefit of this approach is
that it does not need to be tied to the memory manager at all.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This introduces a new way of defining the virtio-balloon device. Instead
of going through the --memory parameter, the idea is to consider balloon
as a standalone virtio device.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The snasphot/restore feature is not working because some CPU states are
not properly saved, which means they can't be restored later on.
First thing, we ensure the CPUID is stored so that it can be properly
restored later. The code is simplified and pushed down to the hypervisor
crate.
Second thing, we identify for each vCPU if the Hyper-V SynIC device is
emulated or not. In case it is, that means some specific MSRs will be
set by the guest. These MSRs must be saved in order to properly restore
the VM.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The watchdog device is created through the "--watchdog" parameter. At
most a single watchdog can be created per VM.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This device operates a single virtq. When the driver offers a descriptor
to the device it is interpreted as a "ping" to indicate that the guest
is alive. A periodic timer fires and if when the timer is fired there
has not been a "ping" from the guest then the device will reset the VM.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Before Virtio-mmio was removed, we passed an optional PCI space address
parameter to AArch64 code for generating FDT. The address is none if the
transport is MMIO.
Now Virtio-PCI is the only option, the parameter is mandatory.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Given the increased amount of output from cloud-hypervisor, this patch
also increased the PIPE_SIZE to 32MB (from 256KB).
Signed-off-by: Bo Chen <chen.bo@intel.com>
This patch prints the complete commandline when launching
cloud-hypervisor. It also prints the details of the `ssh` command if
the command is failing.
Signed-off-by: Bo Chen <chen.bo@intel.com>
Virtio-mmio is removed, now virtio-pci is the only option for virtio
transport layer. We use MSI for PCI device interrupt. While GICv2, the
legacy interrupt controller, doesn't support MSI. So GICv2 is not very
practical for Cloud-hypervisor, we can remove it.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
When shutting down a VM using VFIO, the following error has been
detected:
vfio-ioctls/src/vfio_device.rs:312 -- Could not delete VFIO group:
KvmSetDeviceAttr(Error(9))
After some investigation, it appears the KVM device file descriptor used
for removing a VFIO group was already closed. This is coming from the
Rust sequence of Drop, from the DeviceManager all the way down to
VfioDevice.
Because the DeviceManager owns passthrough_device, which is effectively
a KVM device file descriptor, when the DeviceManager is dropped, the
passthrough_device follows, with the effect of closing the KVM device
file descriptor. Problem is, VfioDevice has not been dropped yet and it
still needs a valid KVM device file descriptor.
That's why the simple way to fix this issue coming from Rust dropping
all resources is to make Linux accountable for it by duplicating the
file descriptor. This way, even when the passthrough_device is dropped,
the KVM file descriptor is closed, but a duplicated instance is still
valid and owned by the VfioContainer.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We turn on that emulation for Windows. Windows does not have KVM's PV
clock, so calling notify_guest_clock_paused results in an error.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
While using the virtio-iommu device involving L2 scenario, and tearing
things down all the way from L2 back to L0 exposed some bad syscalls
that were not part of the authorized list.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that we've written Windows integration tests and the associated
script to launch them, this patch enables the support for Windows tests
in dev_cli.sh, so that we can run it in our Cloud Hypervisor container.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This is a new integration test running Windows as a guest with Cloud
Hypervisor. Once the VM is booted, the test connects to the guest
through SSH and shutdown the VM. If this succeeds, this means the VM
was properly booted to userspace and that the network was functional.
Important to note that because this test generates lots of logs, it
requires a large pipe size for both stdout and stderr.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Introduce a new test that will validate the new option `max_phys_bits`
from the `--cpus` parameter behaves as expected.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
If the user specified a maximum physical bits value through the
`max_phys_bits` option from `--cpus` parameter, the guest CPUID
will be patched accordingly to ensure the guest will find the
right amount of physical bits.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
If the user provided a maximum physical bits value for the vCPUs, the
memory manager will adapt the guest physical address space accordingly
so that devices are not placed further than the specified value.
It's important to note that if the number exceed what is available on
the host, the smaller number will be picked.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to let the user choose maximum address space size, this patch
introduces a new option `max_phys_bits` to the `--cpus` parameter.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The 'GuestAddress::unchecked_add' function has undefined behavior when
an overflow occurs. Its alternative 'checked_add' requires use to handle
the overflow explicitly.
Signed-off-by: Bo Chen <chen.bo@intel.com>
We are now reserving a 256M gap in the guest address space each time
when hotplugging memory with ACPI, which prevents users from hotplugging
memory to the maximum size they requested. We confirm that there is no
need to reserve this gap.
This patch removes the 'reserved gaps'. It also refactors the
'MemoryManager::start_addr' so that it is rounding-up to 128M alignment
when hotplugged memory is allowed with ACPI.
Signed-off-by: Bo Chen <chen.bo@intel.com>
In previous dev_cli.sh, the `uname -m` command will generate
either `x86_64` or `aarch64`, which is inconsistent with the
architectures in the Dockerfile, namely `amd64` and `arm64`.
This will cause some dependancy missing in the docker container
when the docker image is built locally.
This commit fixes this inconsistancy.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
We now try to create a ram region of size 0 when the requested memory
size is the same as current memory size. It results in an error of
`GuestMemoryRegion(Mmap(Os { code: 22, kind: InvalidInput, message:
"Invalid argument" }))`. This error is not meaningful to users and we
should not report it.
Signed-off-by: Bo Chen <chen.bo@intel.com>
A failure appeared in AArch64 musl cross build, after upgrading rust
to v1.47.0. A symbol "strrchr" was missing while linking against
static libfdt.a.
The issue could be caused by missing symbol(s) in new rust toolchain.
This fix pins the rust version in this cross build action to a stable-
enough version. Further upgrade will be done manually after testing.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
"Using a stable sort consumes more memory and cpu cycles. Because values
which compare equal are identical, preserving their relative order (the
guarantee that a stable sort provides) means nothing, while the extra
costs still apply."
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This is a new clippy check introduced in 1.47 which requires the use of
the matches!() macro for simple match blocks that return a boolean.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Instead of having the hypervisor crate embedding Cloud-Hypervisor forks
from the rust-vmm project, it's more appropriate to leave the rust-vmm
references in the hypervisor crate, and have the root Cargo.toml being
patched.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This changeset extends the documentation with the UEFI and Windows
related info. The focus is on providing consumer with a minimum
necessary and proper piece of the information to enter the features
quickly. While UEFI is a cross platform topic, it is a required
prerequisite for the Windows usage.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
The OneRegister literally means "one (arbitrary) register". Just call it
"Register" instead. There is no need to inherit KVM's naming scheme in
the hypervisor agnostic code.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
The goal here is to replace anywhere possible a virtio structure
with a "C, packed" representation by a "C" representation. Some
virtio structures are not expected to be packed, therefore there's
no reason for using the more restrictive "C, packed" representation.
This is important since "packed" representation can still cause
undefined behaviors with Rust 2018.
By removing the need for "packed" representation, we can simplify a
bit of code by deriving the Serialize trait without writing the
implementation ourselves.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Small patch creating a dedicated `block_io_uring_is_supported()`
function for the non-io_uring case, so that we can simplify the
code in the DeviceManager.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The purpose of that step is to make sure each commit builds. The `check`
command is much faster for that purpose.
On my 32-core machine `cargo check --all` takes around 25 seconds while
`cargo build --all` takes around 35 seconds, so that's quite a bit of
time saving there.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Since all unit and integration tests are run inside containers because
they are called from dev_cli.sh, they always run as root. That's why
both unit and integration scripts can be simplified as they don't need
to apply specific capabilities and run cargo tests in a dedicated 'kvm'
group.
Fixes#1683
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Windows will write zero to the IOAPIC version register as part of
probing if the device is there.
Fixes: #1791
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The existing virtio-blk hotplug test is extended by removing and
re-adding the virtio-blk device. This ensures the unplug/re-plug
feature is properly tested.
Fixes#1809
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The existing virtio-net hotplug test is extended by removing and
re-adding the virtio-net device. This ensures the unplug/re-plug
feature is properly tested.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Without the unlink(2) syscall being allowed, Cloud-Hypervisor crashes
when we remove a virtio-vsock device that has been previously added.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
It's important to ensure Cloud-Hypervisor's Git tree is bisectable, and
that's why this commit extends the existing build.yaml Github action.
This will validate that each commit from an incoming pull request can be
built.
Fixes#1808
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The Cloud-Hypervisor fork of the vhost crate contains one small
additional patch compared to the rust-vmm upstream version, meant for
increasing the connection timeout.
This patch is intended to be merged in order to check if it helps our CI
fixing the vhost-user-blk flakes that we've been observing recently.
If it fixes it, we'll submit a similar patch upstream and switch back to
the upstream vhost crate, otherwise we'll simply switch back to the
upstream crate, discarding this patch.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Because of the PCI refactoring that happened in the previous commit
d793cc4da3, the ability to fully remove a
PCI device was altered.
The refactoring was correct, but the usage of a generic function to pass
the same reference for both BusDevice, PciDevice and Any + Send + Sync
causes the Arc::ptr_eq() function to behave differently than expected,
as it does not match the references later in the code. That means we
were not able to remove the device reference from the MMIO and/or PIO
buses, which was leading to some bus range overlapping error once we
were trying to add a device again to the previous range that should have
been removed.
Fixes#1802
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Run loop in hypervisor needs a callback mechanism to access resources
like guest memory, mmio, pio etc.
VmmOps trait is introduced here, which is implemented by vmm module.
While handling vcpuexits in run loop, this trait allows hypervisor
module access to the above mentioned resources via callbacks.
Signed-off-by: Praveen Paladugu <prapal@microsoft.com>
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
A new version of vm-memory was released upstream which resulted in some
components pulling in that new version. Update the version number used
to point to the latest version but continue to use our patched version
due to the fix for #1258
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The PMEM support has an option called "discard_writes" which when true
will prevent changes to the device from hitting the backing file. This
is trying to be the equivalent of "readonly" support of the block
device.
Previously the memory of the device was marked as KVM_READONLY. This
resulted in a trap when the guest attempted to write to it resulting a
VM exit (and recently a warning). This has a very detrimental effect on
the performance so instead make "discard_writes" truly CoW by mapping
the memory as `PROT_READ | PROT_WRITE` and using `MAP_PRIVATE` to
establish the CoW mapping.
Fixes: #1795
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When all refblocks are consumed, the loop looking for the first free
cluster would access the element at refcounts[refcounts.len()], which is
out of bounds. Modify the free cluster search loop to check that the
index is in bounds before accessing it.
Tested-by: Daniel Verkamp <dverkamp@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Commit-Queue: Daniel Verkamp <dverkamp@chromium.org>
(cherry picked from crosvm commit f21572c7187c8beb9c6bfea6446351ae93200d01)
Fixes: #1792
Signed-off-by: Bo Chen <chen.bo@intel.com>
The virtio-balloon change the memory size is asynchronous.
VirtioBalloonConfig.actual of balloon device show current balloon size.
This commit add memory_actual_size to vm.info to show memory actual size.
Signed-off-by: Hui Zhu <teawater@antfin.com>
When the destination mode is physical, the destination field should
only be defined through bits 56-59, as defined in the IOAPIC spec. But
from the APIC specification, the APIC ID is always defined on 8 bits no
matter which destination mode is selected. That's why we always retrieve
the destination field based on bits 56-63.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This patch adds two required dependencies to fuzz/Cargo.toml, and fixes
the building error on the 'block' fuzzer.
Signed-off-by: Bo Chen <chen.bo@intel.com>
Now that Docker images are automatically generated for both amd64 and
arm64 architectures, there's no need to generate the arm64 image locally
on the ARM CI during a CI run. The image should be available from
DockerHub instead.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Write to the exit_evt EventFD which will trigger all the devices and
vCPUs to exit. This is slightly cleaner than just exiting the process as
any temporary files will be removed.
Fixes: #1242
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This commit moves back to the branch "virtio-fs-dev" from virtiofsd, as
we figured the changes needed to use this branch and the requirements
from the new meson build from QEMU.
It updates the container version to ensure the dev_cli.sh script will
rely on the latest container which contains the needed packages.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By fixing the Dockerfile, we have now finalized the automated generation
of the Docker images for both architectures (amd64 and arm64).
Fixes#953
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This patch adds the missing the `iommu` and `id` option for
`VmAddDevice` in the openApi yaml to respect the internal data structure
in the code base. Also, setting the `id` explicitly for VFIO device
hotplug is required for VFIO device unplug through openAPI calls.
Signed-off-by: Bo Chen <chen.bo@intel.com>
According to openAPI specification [1], the format for `integer` types
can be only `int32` or `int64`, unsigned and 8-bits integers are not
supported.
This patch replaces `uint64` with `int64`, `uint32` with `int32` and
`uint8` with `int32`.
[1]: https://swagger.io/specification/#data-types
Signed-off-by: Julio Montes <julio.montes@intel.com>
MsiInterruptGroup doesn't need to know the internal field names of
InterruptRoute. Introduce two helper functions to eliminate references
to irq_fd. This is done similarly to the enable and disable helper
functions.
Also drop the pub keyword from InterruptRoute fields. It is not needed
anymore.
No functional change.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
In order to support both amd64 and arm64, we rely on the TARGETARCH
variable that is passed from the docker buildx command, based on the
platform used to build the container image.
There is no way to rely directly on $(uname -m) to assign a variable
with the correct x86_64 or aarch64 values we're looking for. Both ENV
and ARG don't evaluate the command, which means they see it as a simple
string. Using RUN is the only way to evaluate a command.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to build virtiofsd from the latest build system, the Python
package python3-setuptools is required.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The idea is to rely on this new Github Action to detect when the
Dockerfile is updated after a push to the master branch on the
repository.
Once triggered, this action builds the Docker image for both
linux/amd64 and linux/arm64 platforms, and updates it directly
on Docker Hub.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
According to openAPI specification[1], the format for `integer` types
can be only `int32` or `int64`, unsigned integers are not supported.
This patch replaces `uint64` with `int64`.
[1]: https://swagger.io/specification/#data-types
Signed-off-by: Julio Montes <julio.montes@intel.com>
In order to speed up the Linux boot (so as to avoid it having to scan a
large number of pages) place the MP table directly after the SMBIOS
table if there is sufficient room. The start address of the SMBIOS table
is one of the three (and the largest) location that the MP table can
also be located at.
Before:
[ 0.000399] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT
[ 0.014945] check: Scanning 1 areas for low memory corruption
After:
[ 0.000284] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT
[ 0.000421] found SMP MP-table at [mem 0x000f0090-0x000f009f]
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
There is no point in manually dropping the lock for gsi_msi_routes then
instantly grabbing it again in set_gsi_routes.
Make set_gsi_routes take a reference to the routing hashmap instead.
No functional change intended.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
The MTRR feature was missing from the CPUID, which is causing the guest
to ignore the MTRR settings exposed through dedicated MSRs.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Removing the ISA DMA configurations prevents the kernel from accessing
the port I/O 0x87, which was generating the following warning:
WARN:vmm/src/cpu.rs:378 -- Guest PIO read to unregistered address 0x87
Removing the TELCLOCK configuration prevents the kernel from accessing
the port I/O reserved for the memory manager, which was causing the
following warning:
WARN:vmm/src/memory_manager.rs:289 -- Unexpected offset for accessing
memory manager device: 15
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The ::new() does very little beyond trying to open the /dev/kvm device
so provide a hint to the user about what has gone wrong.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
As discussed in #1707, the `vcpu` thread can be stalled when using
`--serial tty`. To workaround that issue, this patch enforces to resize
the pipe size to 256K when we capture the stdout/stderr of the
cloud-hypervisor child process in the integration tests. Note that the
pipe size (256K) is chosen based on the output size of our integration
tests at this point, which may need to be increased in the future.
Signed-off-by: Bo Chen <chen.bo@intel.com>
By looking at Linux kernel boot time, we identified that a lot of time
was spent registering and unregistering IRQ fds to KVM. This is not
efficient and certainly not a wrong behavior from the Linux kernel,
but rather a problem with the Cloud-Hypervisor's implementation of
MSI-X.
The way to fix this issue is by ensuring the initial conditions are
correct, which means the entire MSI-X vector table must be disabled
and masked. Additionally, each vector must be individually masked.
With these correct conditions, Linux won't start masking interrupt
vectors, and later unmask them since they will be seen as masked from
the beginning. This means the OS will simply have to unmask them when
needed, avoiding the extra operation.
Another aspect of this patch is to prevent Cloud-Hypervisor from
enabling (by registering IRQ fd) all vectors when either the global
'mask' or 'enable' bits are set. Instead, we can simply let the mask()
and unmask() operations take care of it if needed.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Since Cloud-Hypervisor currently support one single PCI bus, we must
reflect this through the MCFG table, as it advertises the first bus and
the last bus available. In this case both are bus 0.
This patch saves quite some time during guest kernel boot, as it
prevents from checking each bus for available devices.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
When created, the IOAPIC entries should be masked, as it is the guest's
responsibility (FW and/or OS) to unmask them if/when necessary.
This patch saves a full round of port I/O writes from the guest to the
IOAPIC, meant for masking the unmasked entries.
Because they're now masked, the entries are not enabled, which means
they are not connected from a KVM perspective, saving from unneeded
registration/unregistration of the irq fds.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
While we figure out the details on how to correctly build virtiofsd from
the latest rebase from the branch "virtio-fs-dev" (which now relies on
QEMU's new build system), let's fix the CI by relying on an older branch
which still relies on the previous build system.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The states of GIC should be part of the VM states. This commit
enables the AArch64 VM states save/restore by adding save/restore
of GIC states.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Currently for AArch64, the GICv3-ITS is tried to be created first
when PCI is not needed, which is unnecessary. This commit fixes
the problem.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
The definition of libc::SYS_ftruncate on AArch64 is different
from that on x86_64. This commit unifies the previously hard-coded
syscall number for AArch64.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
`KVM_GET_REG_LIST` ioctl is needed in save/restore AArch64 vCPU.
Therefore we whitelist this ioctl in seccomp.
Also this commit unifies the `SYS_FTRUNCATE` syscall for x86_64
and AArch64.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Similarly as the VM booting process, on AArch64 systems,
the vCPUs should be created before the creation of GIC. This
commit refactors the vCPU save/restore code to achieve the
above-mentioned restoring order.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Since calling `KVM_GET_ONE_REG` before `KVM_VCPU_INIT` will
result in an error: Exec format error (os error 8). This commit
decouples the vCPU init process from `configure_vcpus`. Therefore
in the process of restoring the vCPUs, these vCPUs can be
initialized separately before started.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
The construction of `GICR_TYPER` register will need vCPU states.
Therefore this commit adds methods to extract saved vCPU states
from the cpu manager.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Unlike x86_64, the "interrupt_controller" in the device manager
for AArch64 is only a `Gic` object that implements the
`InterruptController` to provide the interrupt delivery service.
This is not the real GIC device so that we do not need to save
its states. Also, we do not need to insert it to the device_tree.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
The value of GIC register `GICR_TYPER` is needed in restoring
the GIC states. This commit adds a field in the GIC device struct
and a method to construct its value.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
In AArch64 systems, the state of GIC device can only be
retrieved from `KVM_GET_DEVICE_ATTR` ioctl. Therefore to implement
saving/restoring the GIC states, we need to make sure that the
GIC object (either the file descriptor or the device itself) can
be extracted after the VM is started.
This commit refactors the code of GIC creation by adding a new
field `gic_device_entity` in device manager and methods to set/get
this field. The GIC object can be therefore saved in the device
manager after calling `arch::configure_system`.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This commit adds a function which allows to save RDIST pending
tables to the guest RAM, as well as unit test case for it.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This commit adds the unit test cases for getting/setting the GIC
distributor, redistributor and ICC registers.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This commit implements the `get_device_attr` method for the
`KVM_GET_DEVICE_ATTR` ioctl. This ioctl will be used in retrieving
the GIC status.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This commit moves the GIC-related code to a separate module.
Therefore the implementation of GIC registers can be introduced
to the new module.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Adds 3 more unit test cases for AArch64:
*save_restore_core_regs
*save_restore_system_regs
*get_set_mpstate
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This commit adds methods to save/restore AArch64 vCPU registers,
including:
1. The AArch64 `VcpuKvmState` structure.
2. Some `Vcpu` trait methods of the `KvmVcpu` structure to
enable the save/restore of the AArch64 vCPU states.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This commit ports code from firecracker and refactors the existing
AArch64 code as the preparation for implementing save/restore
AArch64 vCPU, including:
1. Modification of `arm64_core_reg` macro to retrive the index of
arm64 core register and implemention of a helper to determine if
a register is a system register.
2. Move some macros and helpers in `arch` crate to the `hypervisor`
crate.
3. Added related unit tests for above functions and macros.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
virtio-mem device would use 'VIRTIO_MEM_F_ACPI_PXM' to add memory to NUMA
node, which MUST be existed, otherwise it will be assigned to node id 0,
even if user specify different node id.
According ACPI spec about Memory Affinity Structure, system hardware
supports hot-add memory region using 'Hot Pluggable | Enabled' flags.
Signed-off-by: Jiangbo Wu <jiangbo.wu@intel.com>
Use zone.host_numa_node to create memory zone, so that memory zone
can apply memory policy in according with host numa node ID
Signed-off-by: Jiangbo Wu <jiangbo.wu@intel.com>
If after the creation of the self-spawned backend, the VMM cannot create
the corresponding vhost-user frontend, the VMM must kill the freshly
spawned process in order to ensure the error propagation can happen.
In case the child process would still be around, the VMM cannot return
the error as it waits onto the child to terminate.
This should help us identify when self-spawned failures are caused by a
connection being refused between the VMM and the backend.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
When the VMM is terminated by receiving a SIGTERM signal, the signal
handler thread must be able to invoke ioctl(TCGETS) and ioctl(TCSETS)
without error.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Build testing of changes happens on GitHub actions and the integration
tests will build the binary (with different feature flags) again. So
these earlier build operations are just wasted time on the critical
path.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Only use `cloud-hypervisor` when referring to the binary itself and
prefer Cloud Hypervisor when referring to the project.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Some operations complete directly after they have been submitted, which
means they are not submitted asynchronously and therefore they don't
generate any ioevent. This is the reason why we are not processing some
of the completed operations, which leads to some unpredictable
behaviors.
Forcing all io_uring operations submitted to the SQE to be asynchronous
helps simplifying the code as it ensures the completion of every
operation will generate an ioevent, therefore no operation is missed.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that virtio-mem supports reboot, we extend the existing integration
tests to validate the amount of RAM after reboot is the same as before
the reboot, but also that we can still resize down the VM or the memory
zone after the reboot.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The missing syscall rt_sigprocmask(2) was triggered for the musl build
upon rebooting the VM, and was causing the VM to be killed.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to facilitate the memory hotplug for our users, let's enable
the automatic memory onlining in our guest kernel by activating the
kernel config option 'CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE'.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on all the preparatory work achieved through previous commits,
this patch updates the 'hotplugged_size' field for both MemoryConfig and
MemoryZoneConfig structures when either the whole memory is resized, or
simply when a memory zone is resized.
This fixes the lack of support for rebooting a VM with the right amount
of memory plugged in.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Adding a new field to VirtioMemZone structure, as it lets us associate
with a particular virtio-mem region the amount of memory that should be
plugged in at boot.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This patch simplifies the code as we have one single Option for the
VirtioMemZone. This also prepares for storing additional information
related to the virtio-mem region.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Add the new option 'hotplugged_size' to both --memory-zone and --memory
parameters so that we can let the user specify a certain amount of
memory being plugged at boot.
This is also part of making sure we can store the virtio-mem size over a
reboot of the VM.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit gives the possibility to create a virtio-mem device with
some memory already plugged into it. This is preliminary work to be
able to reboot a VM with the virtio-mem region being already resized.
Signed-off-by: Hui Zhu <teawater@antfin.com>
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that e820 tables are created from the 'boot_guest_memory', we can
simplify the memory manager code by adding the virtio-mem regions when
they are created. There's no need to wait for the first hotplug to
insert these regions.
This also anticipates the need for starting a VM with some memory
already plugged into the virtio-mem region.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to differentiate the 'boot' memory regions from the virtio-mem
regions, we store what we call 'boot_guest_memory'. This is useful to
provide the adequate list of regions to the configure_system() function
as it expects only the list of regions that should be exposed through
the e820 table.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Extend the existing test to validate that each NUMA node gets assigned
the right amount of memory after each memory zone has been resized.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that we can resize each memory zone independently, this commit
extends the memory zone related test by validating 'vm.resize-zone'
works correctly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The virtio-mem driver is generating some warnings regarding both size
and alignment of the virtio-mem region if not based on 128MiB:
The alignment of the physical start address can make some memory
unusable.
The alignment of the physical end address can make some memory
unusable.
For these reasons, the current patch enforces virtio-mem regions to be
128MiB aligned and checks the size provided by the user is a multiple of
128MiB.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that virtio-mem device accept a guest NUMA node as parameter, we
retrieve this information from the list of NUMA nodes. Based on the
memory zone associated with the virtio-mem device, we obtain the NUMA
node identifier, which we provide to the virtio-mem device.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Implement support for associating a virtio-mem device with a specific
guest NUMA node, based on the ACPI proximity domain identifier.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
For more consistency and help reading the code better, this commit
renames all 'virtiomem*' variables into 'virtio_mem*'.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By testing manually the memory resizing through virtio-mem, several
missing syscalls have been identified.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Implement a new VM action called 'resize-zone' allowing the user to
resize one specific memory zone at a time. This relies on all the
preliminary work from the previous commits to resize each virtio-mem
device independently from each others.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By adding a new parameter 'id' to the virtiomem_resize() function, we
prepare this function to be usable for both global memory resizing and
memory zone resizing.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
It's important to return the region covered by virtio-mem the first time
it is inserted as the device manager must update all devices with this
information.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the previous code changes, we can now update the MemoryManager
code to create one virtio-mem region and resizing handler per memory
zone. This will naturally create one virtio-mem device per memory zone
from the DeviceManager's code which has been previously updated as well.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In anticipation for resizing support of an individual memory zone,
this commit introduces a new option 'hotplug_size' to '--memory-zone'
parameter. This defines the amount of memory that can be added through
each specific memory zone.
Because memory zone resize is tied to virtio-mem, make sure the user
selects 'virtio-mem' hotplug method, otherwise return an error.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Both MemoryManager and DeviceManager are updated through this commit to
handle the creation of multiple virtio-mem devices if needed. For now,
only the framework is in place, but the behavior remains the same, which
means only the memory zone created from '--memory' generates a
virtio-mem region that can be used for resize.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to anticipate the need for storing memory regions along with
virtio-mem information for each memory zone, we create a new structure
MemoryZone that will replace Vec<Arc<GuestRegionMmap>> in the hash map
MemoryZones.
This makes thing more logical as MemoryZones becomes a list of
MemoryZone sorted by their identifier.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Inject CPUID leaves for advertising KVM HyperV support when the
"kvm_hyperv" toggle is enabled. Currently we only enable a selection of
features required to boot.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Currently we don't need to do anything to service these exits but when
the synthetic interrupt controller is active an exit will be triggered
to notify the VMM of details of the synthetic interrupt page.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The Windows virtio block driver puts multiple data descriptors between
the header and the status footer. To handle this when parsing iterate
over the descriptor chain until the end is reached accumulating the
address and length pairs in a vector. For execution iterate over the
vector and make sequential reads from the disk for each data descriptor.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In anticipation for supporting multiple virtio descriptors, let's make
sure the read/write operations are performed with vectored I/O.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Some of the io_uring setup happens upon activation of the virtio-blk
device, which is initially triggered through an MMIO VM exit. That's why
the vCPU threads must authorize io_uring related syscalls.
This commit ensures the virtio-blk io_uring implementation can be used
along with the seccomp filters enabled.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We observed CI instability for the past couple of days. This
instability is confirmed to be a result of incomplete seccomp
filters. Given the filter on 'virtio_vsock' is recently added and
is missing 'brk', it is likely to be the root cause of the
instability.
Signed-off-by: Bo Chen <chen.bo@intel.com>
Extract common code for adding devices to the PCI bus into its own
function from the VFIO and VIRTIO code paths.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This removes the dependency of the pci crate on the devices crate which
now only contains the device implementations themselves.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Extending the Cloud-Hypervisor CI to allow for testing SGX on a
dedicated machine where special image and kernels are ready.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The new function already checks if the API version is compatible. There
is no need to expose the get_api_version function to code outside
hypervisor crate.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
In our build-script (build.rs), we won't set the environment variable
'BUILD_VERSION' when the 'git describe' command failed (e.g. when the
current source tree does not contain git information). This patch added
a fall back path where the default value of 'BUILD_VERSION' is based on
the 'cloud-hypervisor' crate version.
Fixes: #1669
Signed-off-by: Bo Chen <chen.bo@intel.com>
The goal of this commit is to rename the existing NUMA option 'id' with
'guest_numa_id'. This is done without any modification to the way this
option behaves.
The reason for the rename is caused by the observation that all other
parameters with an option called 'id' expect a string to be provided.
Because in this particular case we expect a u32 representing a proximity
domain from the ACPI specification, it's better to name it with a more
explicit name.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the recent changes regarding how to describe NUMA nodes to the
guest, the documentation related to memory zones and NUMA has been
updated accordingly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Since both --memory-zone and --numa parameters have been updated with
addition and removal of multiple options, this commit updates the
related integration tests to ensure they are still valid.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The way to describe guest NUMA nodes has been updated through previous
commits, letting the user describe the full NUMA topology through the
--numa parameter (or NumaConfig).
That's why we can remove the deprecated and unused 'guest_numa_node'
option.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the previous changes introducing new options for both memory
zones and NUMA configuration, this patch changes the behavior of the
NUMA node definition. Instead of relying on the memory zones to define
the guest NUMA nodes, everything goes through the --numa parameter. This
allows for defining NUMA nodes without associating any particular memory
range to it. And in case one wants to associate one or multiple memory
ranges to it, the expectation is to describe a list of memory zone
through the --numa parameter.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This new option provides a new way to describe the memory associated
with a NUMA node. This is the first step before we can remove the
'guest_numa_node' option from the --memory-zone parameter.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that we have an identifier per memory zone, and in order to keep
track of the memory regions associated with the memory zones, we create
and store a map referencing list of memory regions per memory zone ID.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In anticipation for allowing memory zones to be removed, but also in
anticipation for refactoring NUMA parameter, we introduce a mandatory
'id' option to the --memory-zone parameter.
This forces the user to provide a unique identifier for each memory zone
so that we can refer to these.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Split the block device implementation into code that be used in common
between multiple different virtio device implementations.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In order to simplify the transition to VirtioCommon and to avoid needing
to set empty fields derive Default for VirtioCommon.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Rearrange the code to match other devices which makes it easier to prep
for sharing this between other devices.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Move the if-let for the taps later which makes the earlier activation
code identical to other devices.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Introduce VirtioCommon to help remove duplicated functionality and state
between implementations of VirtioDevice. Initially it is only handling
feature acknowledgement and testing.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This patch ported many tests to the new methodology, where the guest log
will be printed only when the test is failing.
Things to finish in follow-up PRs:
1. Special tests not ported yet include: test_reboot,
test_bzimage_reboot, test_serial_null(), test_serial_tty(),
test_serial_file(), test_virtio_console(), test_console_file(),
and test_simple_launch.
2. Few direct calls to 'Command::new(clh_command("cloud-hypervisor"))',
which is still printing the guest console
Signed-off-by: Bo Chen <chen.bo@intel.com>
Writing some new documentation to help users understand how the guest
memory can be described through Cloud-Hypervisor parameters.
Fixes#1659
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By extending the existing NUMA integration test, this commit validates
the proper distances between NUMA nodes are exposed to the guest.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By introducing the SLIT (System Locality Distance Information Table), we
provide the guest with the distance between each node. This lets the
user describe the NUMA topology with a lot of details so that slower
memory backing the VM can be exposed as being further away from other
nodes.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the NumaConfig which now provides distance information, we can
internally update the list of NUMA nodes with the exact distances they
should be located from other nodes.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By introducing 'distances' option, we let the user describe a list of
destination NUMA nodes with their associated distances compared to the
current node (defined through 'id').
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Extend the existing NUMA integration to validate that specifying CPUs
for each NUMA node gets propagated to the guest.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the list of CPUs related to each NUMA node, Processor Local
x2APIC Affinity structures are created and included into the SRAT table.
This describes which CPUs are part of each node.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Relying on the list of CPUs defined through the NumaConfig, this patch
will update the internal list of CPUs attached to each NUMA node.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Through this new parameter, we give users the opportunity to specify a
set of CPUs attached to a NUMA node that has been previously created
from the --memory-zone parameter.
This parameter will be extended in the future to describe the distance
between multiple nodes.
For instance, if a user wants to attach CPUs 0, 1, 2 and 6 to a NUMA
node, here are two different ways of doing so:
Either
./cloud-hypervisor ... --numa id=0,cpus=0-2:6
Or
./cloud-hypervisor ... --numa id=0,cpus=0:1:2:6
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This new test validates the guest OS can find the NUMA nodes which have
been defined by the user through the CLI, and that the right amount of
memory is associated with each node.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The SRAT table (System Resource Affinity Table) is needed to describe
NUMA nodes and how memory ranges and CPUs are attached to them.
For now it simply attaches a list of Memory Affinity structures based on
the list of NUMA nodes created from the VMM.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the 'guest_numa_node' option, we create and store a list of
NUMA nodes in the MemoryManager. The point being to associate a list of
memory regions to each node, so that we can later create the ACPI tables
with the proper memory range information.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
With the introduction of this new option, the user will be able to
describe if a particular memory zone should belong to a specific NUMA
node from a guest perspective.
For instance, using '--memory-zone size=1G,guest_numa_node=2' would let
the user describe that a memory zone of 1G in the guest should be
exposed as being associated with the NUMA node 2.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Given that ACPI uses u32 as the type for the Proximity Domain, we can
use u32 instead of u64 as the type for 'host_numa_node' option.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
"struct MemoryConfig" has balloon_size but not in MemoryConfig
of cloud-hypervisor.yaml.
This commit adds it.
Signed-off-by: Hui Zhu <teawater@antfin.com>
Let's narrow down the limitation related to mbind() by allowing shared
mappings backed by a file backed by RAM. This leaves the restriction on
only for mappings backed by a regular file.
With this patch, host NUMA node can be specified even if using
vhost-user devices.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Relying on the new option 'host_numa_node' from the 'memory-zone'
parameter, the user can now define which NUMA node from the host
should be used to back the current memory zone.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Since memory zones have been introduced, it is now possible for a user
to specify multiple backends for the guest RAM. By adding a new option
'host_numa_node' to the 'memory-zone' parameter, we allow the guest RAM
to be backed by memory that might come from a specific NUMA node on the
host.
The option expects a node identifier, specifying which NUMA node should
be used to allocate the memory associated with a specific memory zone.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The flag 'mergeable' should only apply to the entire guest RAM, which is
why it is removed from the MemoryZoneConfig as it is defined as a global
parameter at the MemoryConfig level.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The 'cmdline' parameter should not be required as it is not needed when
the 'kernel' parameter is the rust-hypervisor-fw, which means the kernel
and the associated command line will be found from the EFI partition.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Factorize the codepath between simple memory and multiple memory zones.
This simplifies the way regions are memory mapped, as everything relies
on the same codepath. This is performed by creating a memory zone on the
fly for the specific use case where --memory is used with size being
different from 0. Internally, the code can rely on memory zones to
create the memory regions forming the guest memory.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
After the introduction of user defined memory zones, we can now remove
the deprecated 'file' option from --memory parameter. This makes this
parameter simpler, letting more advanced users define their own custom
memory zones through the dedicated parameter.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
User defined memory regions can now support being snapshot and restored,
therefore this commit removes the restrictions that were applied through
earlier commit.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
By factorizing a lot of code into create_ram_region(), this commit
achieves the simplification of the restore codepath. Additionally, it
makes user defined memory zones compatible with snapshot/restore.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
First thing, this patch introduces a new function to identify if a file
descriptor is linked to any hard link on the system. This can let the
VMM know if the file can be accessed by the user, or if the file will
be destroyed as soon as the VMM releases the file descriptor.
Based on this information, and associated with the knowledge about the
region being MAP_SHARED or not, the VMM can now decide to skip the copy
of the memory region content. If the user has access to the file from
the filesystem, and if the file has been mapped as MAP_SHARED, we can
consider the guest memory region content to be present in this file at
any point in time. That's why in this specific case, there's no need for
performing the copy of the memory region content into a dedicated file.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Let's not assume that a backing file is going to be the result from
a snapshot for each memory region. These regions might be backed by
a file on the host filesystem (not a temporary file in host RAM), which
means they don't need to be copied and stored into dedicated files.
That's why this commit prepares for further changes by introducing an
optional PathBuf associated with the snapshot of each memory region.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
There will be some cases where the implementation of the snapshot()
function from the Snapshottable trait will require to modify some
internal data, therefore we make this possible by updating the trait
definition with snapshot(&mut self).
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Adding a small test to validate that user defined memory zones work as
expected when using --memory-zone option.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In case the memory size is 0, this means the user defined memory
zones are used as a way to specify how to back the guest memory.
This is the first step in supporting complex use cases where the user
can define exactly which type of memory from the host should back the
memory from the guest.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In anticipation for the need to map part of a file with the function
create_ram_region(), it is extended to accept a file offset as argument.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In case the provided backing file is an actual file and not a directory,
we should not truncate it, as we expect the file to already be the right
size.
This change will be important once we try to map the same file through
multiple memory mappings. We can't let the file be truncated as the
second mapping wouldn't work properly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Introducing a new CLI option --memory-zone letting the user specify
custom memory zones. When this option is present, the --memory size
must be explicitly set to 0.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
It is otherwise seems to be able to cause resource conflicts with
Windows APCI_HAL. The OS might do a better job on assigning resources
to this device, withouth them to be requested explicitly. 0xcf8 and
0xcfc are only what is certainly needed for the PCI device enumeration.
Signed-off-by: Anatol Belski <anatol.belski@microsoft.com>
Use of backing file is deprecated hence use the `hugepages` field.
Also use the `boot` field for specifying number of cpus
Signed-off-by: Amey Narkhede <ameynarkhede02@gmail.com>
We may need to store hypervisor speciific data to the VM. This support is
needed for Microsoft hyperv implementations. This patch introduces two
new definitions to Vm trait and implements for KVM.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Problem with the previous solution was that Cleanup stage was not
executed when previous stages failed. We fix this by adding a post
section that executes always after the Aarch64 build completed, no
matter if it has failed, succeeded or been aborted.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
"debug!" marco is used in virtio-devices/src/epoll_helper.rs. When"-vvv"
and "--log-file" option was specified, the missing "SYS_write" rule
caused a "bad system call" crash.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Cleanup of the Aarch64 machine can't be done as part of the parallel
stage as this is often skipped. When the build is aborted because
another parallel stage failed, the post actions are simply not
performed. That's why we need a dedicated stage, out of the parallel
ones, to cleanup the Aarch64 machine.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Use updated --memory option parameters. The field `size` needs
M/G suffix. Without the suffix cloud-hypervisor panics at
src/main.rs:353
Also the use of backing file is deprecated so use `shared` field
Signed-off-by: Amey Narkhede <ameynarkhede02@gmail.com>
As we never join the spawned virtio-devices worker threads, the error
returned from each worker thread is lost. For now, we simply print out
the error from each worker thread.
Fixes: #1551
Signed-off-by: Bo Chen <chen.bo@intel.com>
From the experiments of running integration tests on my local machine,
auditd occationally reported the 'brk' syscall is needed for the
'virtio-rng' worker thread.
Signed-off-by: Bo Chen <chen.bo@intel.com>
This patch adds the seccomp filter list for the virtio_net thread, while
the list was already added for the virtio_net_ctl thread.
Partially fixes: #925
Signed-off-by: Bo Chen <chen.bo@intel.com>
The current seccomp filter for virtio-net is actually for the worker
thread 'virtio_net_ctl' (not the actual worker thread
'virtio_net'). This patch introduces changes to distinguish those two
worker threads and seccomp filters.
Signed-off-by: Bo Chen <chen.bo@intel.com>
We recently added the ability for the gnu and musl workers to retry if
integration test were not passing, relying on some simple Jenkins
options. Unfortunately, this is not working as expected as the retries
never pass either. The suspected reason is the machine itself, which
might be scheduled on some specific hardware, which makes our VMs more
error prone.
Bottom line, on a faulty machine, the tests will always fail, therefore
there is no added value in retrying on the same machine.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Some OS might check for duplicates and bail out, if it can't create a
distinct mapping. According to ACPI 5.0 section 6.1.12, while _UID is
optional, it becomes required when there are multiple devices with the
same _HID.
Signed-off-by: Anatol Belski <ab@php.net>
Using the Rust Barrier mechanism, this patch forces each virtio device
to acknowledge they've been correctly paused before going further.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
If the build was aborted then the dev_cli.sh code that is responsible
for changing the file ownership will not get run. This results in the
failure to delete some of the files in the workspace.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The seccomp filters specific to the virtio-net threads must contain
dup() syscall now that we ported the epoll code to the EpollHelper.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Migrate virtio-net and vhost-user-net control queue to EpollHelper so
as to remove code duplication.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This required a bit of rearranging as it is not possible to call
prepare_daemon() inside a catch_unwind{} block.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Migrate all vhost-user devices to EpollHelper so as to remove code that
is duplicated between multiple virtio devices.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now the testing aspects are run inside a panic handler block rather than
inside a credibility TestBlock. If the test fails then the output from
the cloud-hypervisor binary is then presented.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This reduces the complexity of the test slightly. The PCI BDFs in the L1
needed changing as the block devices come before the network ones.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Migrate to EpollHelper so as to remove code that is duplicated between
multiple virtio devices.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Instead of passing only the event type through the handle_event()
callback, we make the trait slightly more generic by providing the
epoll event to each virtio device implementation.
This is particularly useful for vsock as it will need the event set.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Migrate to EpollHelper so as to remove code that is duplicated between
multiple virtio devices.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The seccomp filters specific to the virtio-console thread must contain
dup syscall now that we ported the epoll code to the EpollHelper.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Migrate to EpollHelper so as to remove code that is duplicated between
multiple virtio devices.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The brk syscall is not always called as the system might not need it.
But when it's needed from the API thread, this causes the thread to
terminate as it is not part of the authorized list of syscalls.
This should fix some sporadic failures on the CI with the musl build.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Even if the vhost-user-net device did implement all currently-defined
features, it would be very short-sighted to use ::all(), because if a
new feature was defined later, the device would start claiming to
implement it even though it didn't.
More practically, claiming to implement all features breaks using QEMU
with the cloud-hypervisor vhost-user-net backend, because QEMU will
negotiate VHOST_USER_PROTOCOL_F_SLAVE_REQ, and then break when the
communication channel isn't actually set up.
I wasn't sure exactly which features the backend should claim to
implement, though. Definitely MQ, and I'm fairly certain none of the
features I've ommitted are implemented. But I'm not sure about
REPLY_ACK. As far as I can tell it should be implemented entirely by
the vhost crate, with no cooperation required from the vhost-user-net
backend itself, so there should be no reason to let a frontend use it
if it wants to. But despite this, neither vhost-user-fs nor
vhost-user-blk claims to implement it.
Signed-off-by: Alyssa Ross <hi@alyssa.is>
Rather than using a credible TestBlock to capture the test assertions
instead use a catch_unwind block to catch the panic and turn
it into a Result<>.
If block panicked or the child binary had non-zero exit then, and only
then, print out the child output.
This results in a clearer test output with no interleaving.
Currently only test_counters is ported to this methodology to
demonstrate its use.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
`cargo test` runs every test in its own process and by default will
collect the output from the test and only print it if the test fails.
`--nocapture` instead prints any output from the test immediately and
does not collate it - this leads to interleaving.
`--nocapture` might be useful for local test debugging but is not
something that is helpful, and is instead unhelpful for our CI as a
whole.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Now that io-uring crate has been stabilized, let's move to the latest
release 0.4.0 from crates.io.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Migrate to EpollHelper so as to remove code that is duplicated between
multiple virtio devices.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Packaging the cloud-hyperviosr binary is useful for enterprise use.
This patch adds the spec file for x86_64. Aarch64 support is not added
yet.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Both gnu and musl workers will retry integration tests up to 3 times if
they fail. This should give us a better pass rate, without having to
restart the entire build every time a single glitch happens.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Add mprotect to API thread rules. Prevent the VMM is
killed when it is used.
Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
This fixes `cargo vendor` throwing an error
```
$ cargo vendor
error: failed to sync
Caused by:
found duplicate version of package `vfio-bindings v0.2.0` vendored from two sources:
source 1: https://github.com/rust-vmm/vfio-bindings#f08cbcbf
source 2: registry `https://github.com/rust-lang/crates.io-index`
```
Both sources are indeed same, the conflict is only cause by the
different URLs.
Signed-off-by: Anatol Belski <ab@php.net>
Based on the way KVM_GET_MSRS and KVM_SET_MSRS work, both function are
very unlikely to fail, as they simply stop looping through the list of
MSRs as soon as getting or setting one fails. This is causing some
issues with the snapshot/restore feature, as on some platforms, we only
save a subset of the list of MSRs, leading to unproper way of saving the
VM.
The way to address this issue is by checking the number of MSRs get/set
matches the expected amount from the list. In case it does not match, we
simply ignore the failing MSR and continue getting/setting the rest of
the list. By doing this by iterations, we end up getting/setting as many
MSRs as the platform can support.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Turns out that "when" blocks are permitted but not effectual in "post"
sections so instead use a script behaviour to make the message
conditional on build change.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Migrate to EpollHelper so as to remove code that is duplicated between
multiple virtio devices.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Currently any messages generated during the worker thread are not
shown anywhere as the thread is never join()ed on. Instead output the
error immediately.
For now only cover the subset where the work to port to EpollHandler
clashed with the seccomp filtering for virtio devices.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
We no longer download focal-server-cloudimg-arm64.img, checking it
lead to a failure if "work_loads" folder is new.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Migrate to EpollHelper so as to remove code that is duplicated between
multiple virtio devices.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Migrate to EpollHelper so as to remove code that is duplicated between
multiple virtio devices.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Increase the number of open files limit for the sandboxed process to the
maximum allowed in the system. The maximum is obtained by reading the
/proc/sys/fs/nr_open sysctl file, and the setting is done using the setrlimit
syscall. Failure to read or parse the nr_open file, or to set the rlimit
results in a panic.
Signed-off-by: Ricardo Koller <ricarkol@gmail.com>
This commit enables the test case for testing the basic function
of virtio_vsock (i.e. without the hotplug).
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This patch added the seccomp_filter module to the virtio-devices crate
by taking reference code from the vmm crate. This patch also adds
allowed-list for the virtio-block worker thread.
Partially fixes: #925
Signed-off-by: Bo Chen <chen.bo@intel.com>
This patch propagates the SeccompAction value from main to the
Vm struct constructor (i.e. Vm::new_from_memory_manager), so that we can
use it to construct the DeviceManager and CpuManager struct for
controlling the behavior of the seccomp filters for vcpu/virtio-device
worker threads.
Signed-off-by: Bo Chen <chen.bo@intel.com>
This patch extends the CLI option '--seccomp' to accept the 'log'
parameter in addition 'true/false'. It also refactors the
vmm::seccomp_filters module to support both "SeccompAction::Trap" and
"SeccompAction::Log".
Fixes: #1180
Signed-off-by: Bo Chen <chen.bo@intel.com>
This patch replaces the usage of 'SeccompLevel' with 'SeccompAction',
which is the first step to support the 'log' action over system
calls that are not on the allowed list of seccomp filters.
Signed-off-by: Bo Chen <chen.bo@intel.com>
This commit enables the `api_create_boot` case in the integration
test as the test for the Cloud Hypervisor API server functionality.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
By adding a new io_uring feature gate, we let the user the possibility
to choose if he wants to enable the io_uring improvements or not.
Since the io_uring feature depends on the availability on recent host
kernels, it's better if we leave it off for now.
As soon as our CI will have support for a kernel 5.6 with all the
features needed from io_uring, we'll enable this feature gate
permanently.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Instead of just logging error messages but continue the processing of
the queues, this patch returns errors right away. This allows for a
quicker detection of an error happening on the virtqueue.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In case the host supports io_uring and the specific io_uring options
needed, the VMM will choose the asynchronous version of virtio-blk.
This will enable better I/O performances compared to the default
synchronous version.
This is also important to note the VMM won't be able to use the
asynchronous version if the backend image is in QCOW format.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This introduces a new version of virtio-blk device. The default
virtio-blk provides synchronous processing of the queues, while this
new version relies on io_uring from the host kernel to provide an
asynchronous processing of the queues.
This new asynchronous version provides a huge performance improvement
compared to the default synchronous version.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Creates a dedicated function relying on io_uring crate to execute
io_uring specific requests.
Also creates a function for checking io_uring support on the host.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
PR#1511 introduced a `flock` command in order to let AArch64 CI
can be executed with multiple executors. However the command
```
(
echo "try to lock $WORKLOADS_DIR folder and update"
flock -x 12 && update_workloads
)
```
will introduce an abnormal behavior: If any error happened in
function `updated_workloads`, the sub-shell opened by the pair of
parentheses will be killed instead of the main shell, which is not
right.
This commit fixes this abnormal execution behaviour.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Cloud Hypervisor allows either the serial or virtio console to output to
TTY, but TTY input is pushed to both.
This is not correct. When Linux guest is configured to spawn TTYs on
both ttyS0 and hvc0, the user effectively issues the same commands twice
in different TTYs.
Fix this by only direct input to the one choice that is using host side
TTY.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Lock "work_loads" folder when one job is syncing files. If another job
arrives, wait until the lock is released.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
The command which is mentioned in the README is wrong. We have two
cofigs one for x86_64 and another for aarh64. Previously it was a single
config. After adding the configs the read me was not modified. This
patch fixes the issue.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
In order to follow the virtio-fs development, we rely on the latest
development branch regarding the virtio-fs daemon. It will be now build
from virtio-fs-dev branch from the virtio-fs maintainers repository.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Move to latest vhost version, as it contains the updated list of
vhost-user protocol features along with the updated list of virtio-fs
slave commands.
This will allow Cloud-Hypervisor to work with latest virtiofsd binary.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit fixes an "Bad syscall" error when shutting down the VM
on AArch64 by adding the SYS_unlinkat syscall to the seccomp
whitelist.
Signed-off-by: Jianyong Wu <jianyong.wu@arm.com>
EpollHelper allows the removal of much duplicated loop handling code and
instead the device specific even handling is delegated via an
implementation of EpollHelperHandler.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This is required for implementing virtio-net as the epoll RawFd must be
assigned into the NetQueuePair.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Ensure that the width of the I/O port is correctly set to 32-bits in the
generic address used for the X_PM_TMR_BLK. Do this by type
parameterising GenericAddress::io_port_address() fuction.
TEST=Boot with clocksource=acpi_pm and observe no errors in the dmesg.
Fixes: #1496
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Port virtio-block device to the new EpollHelper. This required moving
the queue EventFd ownership to BlockEpollHandler.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This is a helper for implementing the worker thread for virtio devices
and in particular handles special behaviour for pause and kill events.
The device specific event handling (for the queues themselves) is
delegated to a method invoked on a new EpollHelperHandler trait. This
method is passed the event as well as the EpollHelper so that it can
operate on the handler in order to manage events itself (required for
virtio-net.)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This is a counter exposed via an I/O port that runs at 3.579545MHz. Here
we use a hardcoded I/O and expose the details through the FADT table.
TEST=Boot Linux kernel and see the following in dmesg:
[ 0.506198] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add some documentation regarding the snapshot/restore feature which was
introduced earlier, explaining how the feature works and how to use it
properly.
Fixes#1490
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This bit should be set for the last 224us of each second. To be able to
fill this bit we need to use clock_gettime() to get the nanoseconds.
Also set the frequency bit to indicate we update at 32kHz.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
We store the device passthrough handler, so we should use it through our
internal API and only carry the passed through device configuration.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Shrink GICDevice trait to contain hypervisor agnostic API's only, which
are used in generating FDT.
Move all KVM specific logic into KvmGICDevice trait.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
This commit enables some mmio-related integration test cases on
AArch64, including:
* some vhost_user test cases
* virtio-blk test cases
* pmem test cases
Also this commit contains a bug fix in creating virtio-blk device.
Previously, when creating the FDT, the virtio-blk device was
labeled in the reverse order of address allocation.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Make set_gsi_routing take a list of IrqRoutingEntry. The construction of
hypervisor specific structure is left to set_gsi_routing.
Now set_gsi_routes, which is part of the interrupt module, is only
responsible for constructing a list of routing entries.
This further splits hypervisor specific code from hypervisor agnostic
code.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
That function is going to return a handle for passthrough related
operations.
Move create_kvm_device code there.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Add the basic infrastructure for fuzzing along with a qcow fuzzer ported
from crosvm and adapted to our code.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The only driver writable field in the virtio-block specification is the
writeback one. Check that the offset being written to is for that field
and update it.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The only driver writable field in the virtio-block specification is the
writeback one. Check that the offset being written to is for that field
and update it.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add a helper function to share code between implementations that can use
a slice accessible data structure for configuration data.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Remove the write_config() implementations that only generate a warning
as that is now done at the VirtioDevice level.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Not every virtio device has any config fields that can be read and most
have none that can be written to.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The retry order to create virtual GIC is GICv3-ITS, GICv3 and GICv2.
But there was not log message to show what was finally created.
The log messages also mute the warning for unused "log" crate.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
With recent changes (#1030#1084), the '/vm.remove-device' API is no
longer restricted to remove VFIO PCI devices. It now supports (almost)
all hotpluggable devices, e.g. disk/net/pmem/fs/vsock.
Signed-off-by: Bo Chen <chen.bo@intel.com>
It returns an hypervisor object depending on which hypervisor is
configured. Currently it only supports KVM.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
SGX expects the EPC region to be reported as "reserved" from the e820
table. This patch adds a new entry to the table if SGX is enabled.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The support for SGX is exposed to the guest through CPUID 0x12. KVM
passes static subleaves 0 and 1 from the host to the guest, without
needing any modification from the VMM itself.
But SGX also relies on dynamic subleaves 2 through N, used for
describing each EPC section. This is not handled by KVM, which means
the VMM is in charge of setting each subleaf starting from index 2
up to index N, depending on the number of EPC sections.
These subleaves 2 through N are not listed as part of the supported
CPUID entries from KVM. But it's important to set them as long as index
0 and 1 are present and indicate that SGX is supported.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Instead of passing the GuestMemoryMmap directly to the CpuManager upon
its creation, it's better to pass a reference to the MemoryManager. This
way we will be able to know if SGX EPC region along with one or multiple
sections are present.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The SGX EPC region must be exposed through the ACPI tables so that the
guest can detect its presence. The guest only get the full range from
ACPI, as the specific EPC sections are directly described through the
CPUID of each vCPU.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on the presence of one or multiple SGX EPC sections from the VM
configuration, the MemoryManager will allocate a contiguous block of
guest address space to hold the entire EPC region. Within this EPC
region, each EPC section is memory mapped.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Introducing the new CLI option --sgx-epc along with the OpenAPI
structure SgxEpcConfig, so that a user can now enable one or multiple
SGX Enclave Page Cache sections within a contiguous region from the
guest address space.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Extract the code that is used by vhost_user_block from the
virtio-devices crate to remove the dependencies on unrequired
functionality such as the virtio transports.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This commit adds required environment configurations to the
`dev_cli.sh` and a Jenkins stage to enable AArch64 binary
building using musl toolchain.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
As the current AArch64 integration test builds kernel every time,
which would take unnecessary time in CI and therefore not ideal.
This commit refactors the AArch64 kernel building strategy to:
1. Keep the Linux kernel source code directory instead of deleting
it everytime after the kernel is built.
2. In integration test script, check if the Linux kernel source
code directory exists. If so use `git fetch && git checkout -f` to
keep the source code always updated, else run `git clone` to get
the source code.
3. Copy config file in and then compile the kernel.
Fixes: https://github.com/cloud-hypervisor/cloud-hypervisor/issues/1444
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This simplies some of the handling for PCI BARs particularly with
respect to snapshot and restore. No attempt has been made to handle the
64-bit bar handling in a different manner to that which was used before.
Fixes: #1153
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In this commit we saved the BDF of a PCI device and set it to "devid"
in GSI routing entry, because this field is mandatory for GICv3-ITS.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
The type is now hypervisor::Vm. Switch from KVM specific name vm_fd to a
generic name just like 8186a8eee6 ("vmm: interrupt: Rename vm_fd").
No functional change.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Under high load, the VM might take some time to hotplug the disk after
the hotplug command has been issued. For this reason, let's put a 10s
sleep before checking for the presence of the new disk.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We want to give the time to the nested VM to be fully ready before we
check it's correctly setup. This involves 3 layers of virtualization
when running the CI on Azure, which added to the high load happening
because of the parallelization, adds up to the start up time.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The virtio configuration structures have been slightly modified between
5.6-rc4 and 5.8-rc4, forcing the virtio-iommu device to be updated
accordingly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Updating the kernel from 5.6-rc4 to 5.8-rc4 allows us to remove the
dependency on both virtio-vsock and virtio-mem patches as they are now
part of the upstream kernel. We're still carrying virtio-iommu and
virtio-fs patches.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Move the definition of RawFile from virtio-devices crate into qcow
crate. All the code that consumes RawFile also already depends on the
qcow crate for image file type detection so this change breaks the
need for the qcow crate to depend on the very large virtio-devices
crate.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
As compiling without acpi (implied by mmio) means that the VM will
terminate on i8042 reset we cannot test the reboot.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Now that vhost_user_net crate does not depend on the virtio-devices
crate it is no longer compiled differently based on the mmio or pci
features.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Move NetQueuePair and the related NetCounters into the net_util crate.
This means that the vhost_user_net crate now no longer depends on
virtio-devices and so does not depend on the pci, qcow or other similar
crates. This significantly simplifies the build chain for this backend.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
By moving the code for opening the two RX and TX queues into a shared
location we are starting to remove the requirement for the
vhost-user-net backend to depend on the virtio-devices crate which in of
itself depends on many other crates that are not necessary for the
backend to function.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Rather than use an embedded String inside the MultiQueueSupport error
value use two different values to differentiate the two cases.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
By moving the code for opening the TAP device into a shared location we
are starting to remove the requirement for the vhost-user-net backend to
depend on the virtio-devices crate which in of itself depends on many
other crates that are not necessary for the backend to function.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
It gets bubbled all the way up from hypervsior crate to top-level
Cargo.toml.
Cloud Hypervisor can't function without KVM at this point, so make it
a default feature.
Fix all scripts that use --no-default-features.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Check if increasing the time after the VM is spawned help with getting
more stable numbers for the base PSS.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit store balloon size to MemoryConfig.
After reboot, virtio-balloon can use this size to inflate back to
the size before reboot.
Signed-off-by: Hui Zhu <teawater@antfin.com>
The implementation of the virtio-balloon was slightly wrong as it was
generating the GPA (Guest Physical Address) from the PFN (Page Frame
Number) which was a u32. That means the GPA was created as a u32, and
later a cast was done to extend it to a u64 type. Unfortunately, by
doing so, the GPA was wrong if the value was supposedly more than 32
bits.
That's why the PFN is casted into a u64 before the GPA is generated,
which creates the GPA on 64 bits directly.
Additionally, this patch simplifies the process_queue() function,
relying on multiple vm-memory helpers.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This commit adds a Jenkins stage for AArch64 integration test,
and the test is carried out using the GNU toolchain.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
The compiled AArch64 linux kernel by running `make` is in PE format
instead of vmlinux, vmlinux.pvh and bzImage format. Therefore we
need to add integration test for this case.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
This commit adds supporting components and code for enabling the
AArch64 integration tests, including:
1. A Linux kernel config file to build kernel on AArch64 machines.
2. Refactoring the `run_integration_test.sh` to architecture
specific scripts for readability.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Remove the vmm dependency from vhost_user_block and vhost_user_net where
it was existing to use config::OptionParser. By moving the OptionParser
to its own crate at the top-level we can remove the very heavy
dependency that these vhost-user backends had.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Refactored the construction of KVM IOCTL rules for Seccomp.
Separating the rules by architecture can reduce the risk of bugs and
attacks.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
In order to move the hypervisor specific parts of the VM exit handling
path, we're defining a generic, hypervisor agnostic VM exit enum.
This is what the hypervisor's Vcpu run() call should return when the VM
exit can not be completely handled through the hypervisor specific bits.
For KVM based hypervisors, this means directly forwarding the IO related
exits back to the VMM itself. For other hypervisors that e.g. rely on the
VMM to decode and emulate instructions, this means the decoding itself
would happen in the hypervisor crate exclusively, and the rest of the VM
exit handling would be handled through the VMM device model implementation.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Fix test_vm unit test by using the new abstraction and dropping some
dead code.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
The code is purely for maintaining an internal counter. It is not really
tied to KVM.
No functional change.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
There are several dependencies that need updating so update them
manually rather than relying on dependabot. This will reduce the load on
the CI.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The binary is still built in the same location but the source code and
the dependencies for it come from the vhost_user_block crate itself.
The binary will be built with:
`cargo build --all --bin vhost_user_block` or just `cargo build --all`
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The binary is still built in the same location but the source code and
the dependencies for it come from the vhost_user_net crate itself.
The binary will be built with:
`cargo build --all --bin vhost_user_net` or just `cargo build --all`
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
With vhost_user_fs binary moved to its own crate the dependencies in the
top level can be trimmed significantly.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The binary is still built in the same location but the source code and
the dependencies for it come from the vhost_user_fs crate itself.
The binary will be built with:
`cargo build --all --bin vhost_user_fs` or just `cargo build --all`
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In preparation for splitting the binaries into their own crates start
building all the binaries in the workspace as part of the integration
testing suite.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In preparation for splitting the binaries into their own crates start
building all the binaries in the workspace when running the build
command inside the container.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In preparation for splitting the binaries into their own crates start
building all the binaries in the workspace when doing a build as part of
the GitHub actions.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The _fd suffix is KVM specific. But since it now point to an hypervisor
agnostic hypervisor::Vm implementation, we should just rename it vm.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The _fd suffix is KVM specific. But since it now point to an hypervisor
agnostic hypervisor::Vm implementation, we should just rename it vm.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The fd naming is quite KVM specific. Since we're now using the
hypervisor crate abstractions, we can rename those into something more
readable and meaningful.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The fd naming is quite KVM specific. Since we're now using the
hypervisor crate abstractions, we can rename those into something more
readable and meaningful.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The fd naming is quite KVM specific. Since we're now using the
hypervisor crate abstractions, we can rename those into something more
readable and meaningful. Like e.g. vcpu or vm.
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Now that our CI has transitioned from ClearLinux to Ubuntu images
exclusively, let's update the documentation to refer to Ubuntu images
instead of ClearLinux's ones.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that the project relies on Ubuntu images, the documentation related
to the creation of a custom image has been updated. It's important to
note this procedure could be applied to any other distribution, given
that the package manager's commands would be slightly different.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to differentiate tests that can be run in parallel versus
tests that must be run on their own, we move all tests into dedicated
modules.
The point is to avoid glitches in results that can be caused by the fact
that other tests (hence VMs) are running at the same time.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Split the generic virtio code (queues and device type) from the
VirtioDevice trait, transport and device implementations.
This also simplifies the feature handling in vhost_user_backend as the
vm-virtio crate is no longer has any features.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Move the CI to rely entirely on Ubuntu cloud images. It's worth noting
that both QCOW2 and RAW images from Ubuntu Focal Fossa have been
modified to include the tools needed from integration tests.
This means fio, iperf, iperf3, netcat and socat have been added to the
image. The snapd package have been fully removed as it was expecting the
support for squashfs (not present when using our own kernel from direct
kernel boot), which was causing some failures, and was preventing
cloud-init from terminating properly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Updating the cloud-init files related to Ubuntu images is needed to be
able to use Ubuntu images for running all integration tests.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
With QCOW disk images the space needed is greater than the size of the
iamge as any "zero" blocks in the image are allocated when they are
touched making the image bigger.
Here we add a threshold of 6GiB with added debugging messages to ensure
that there is sufficient disk space to run the tests.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This update of Dockerfile will add support to the AArch64
integration tests and musl building, including:
1. Installation of the missing `setcap` binary on AArch64.
On AArch64, `setcap` binary should be installed via `libcap2-bin`.
This binary is needed in the integration test.
2. Related support for the cloud-hypervisor binary building by
musl toolchain.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Extend the existing integration test test_snapshot_restore by testing
with more than one vCPU.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
On x86 architecture, we need to save a list of MSRs as part of the vCPU
state. By providing the full list of MSRs supported by KVM, this patch
fixes the remaining snapshot/restore issues, as the vCPU is restored
with all its previous states.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Add a new function to the hypervisor trait so that the caller can
retrieve the list of MSRs supported by this hypervisor.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Some vCPU states such as MP_STATE can be modified while retrieving
other states. For this reason, it's important to follow a specific
order that will ensure a state won't be modified after it has been
saved. Comments about ordering requirements have been copied over
from Firecracker commit 57f4c7ca14a31c5536f188cacb669d2cad32b9ca.
This patch also set the previously saved VCPU_EVENTS, as this was
missing from the restore codepath.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The logic can be shared among hypervisor implementations.
The 'static bound is used such that we don't need to deal with extra
lifetime parameter everywhere. It should be okay because we know the
entry type E doesn't contain any reference.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
This trait contains a function which produces a interrupt routing entry.
Implement that trait for KvmRoutingEntry and rewrite the update
function.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
The observation is only the route entry is hypervisor dependent.
Keep a definition of KvmMsiInterruptManager to avoid too much code
churn.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
The observation is that only the route field is hypervisor specific.
Provide a new function in blanket implementation. Also redefine
KvmRoutingEntry with RoutingEntry to avoid code churn.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
The observation is that the GSI hashmap remains untouched before getting
passed into the MSI interrupt manager. We can create that hashmap
directly in the interrupt manager's new function.
The drops one import from the interrupt module.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
The OVMF firmware loops around looking for an entry marking the end of
the table. Without this entry processing the tables is an infinite loop.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Taken from crosvm: 44336b913126d73f9f8d6854f57aac92b5db809e and adapted
for Cloud Hypervisor.
This is basic and incomplete support but Linux correctly finds the DMI
data based on this:
root@clr-c6ed47bc1c9d473d9a3a8bddc50ee4cb ~ # dmesg | grep -i dmi
[ 0.000000] DMI: Cloud Hypervisor cloud-hypervisor, BIOS 0
root@clr-c6ed47bc1c9d473d9a3a8bddc50ee4cb ~ # dmesg | grep -i smbio
[ 0.000000] SMBIOS 3.2.0 present.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Update the API documentation to reflect that the hotplug APIs return
data about the device as well as the newly added /vm.counters API.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This is a hash table of string to hash tables of u64s. In JSON these
hash tables are object types.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Initially the licensing was just Apache-2.0. This patch changes
the licensing to dual license Apache-2.0 OR BSD-3-Clause
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Each virtio thread was reading/draining the pause_evt pipe when
detecting the associated event. Problem is, when a virtio device has
multiple threads, they all share the same pause_evt pipe, which can
prevent some threads from receiving the event. If the first thread to
catch the event is quickly clearing the pipe, some other threads might
simply miss the event and they will not enter the "paused" state as
expected.
This is a behavior that was spotted with virtio-net as it usually uses
2 threads by default (1 for TX/RX queues and 1 for the control queue).
The way to solve this issue is by letting each thread drain the pipe
during the resume codepath, that is after the thread has been unparked.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Because we don't want the guest to miss any event triggered by the
emulation of devices, it is important to resume all vCPUs before we can
resume the DeviceManager with all its associated devices.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We need consistency between pause/resume and snapshot/restore
operations. The symmetrical behavior of pausing/snapshotting
and restoring/resuming has been introduced recently, and we must
now ensure that no matter if we're using pause/resume or
snapshot/restore features, the resulting VM should be running in
the exact same way.
That's why the vCPU state is now stored upon VM pausing. The snapshot
operation being a simple serialization of the previously saved state.
The same way, the vCPU state is now restored upon VM resuming. The
restore operation being a simple deserialization of the previously
restored state.
It's interesting to note that this patch ensures time consistency from a
guest perspective, no matter which clocksource is being used. From a
previous patch, the KVM clock was saved/restored upon VM pause/resume.
We now have the same behavior for TSC, as the TSC from the vCPUs are
saved/restored upon VM pause/resume too.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Instead of calling the resume() function from the CpuManager, which
involves more than what is needed from the shutdown codepath, and
potentially ends up with a deadlock, we replace it with a subset.
The full resume operation is reserved for a VM that has been paused.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We want each Vcpu to store the vCPU state upon VM pausing. This is the
reason why we need to explicitly implement the Pausable trait for the
Vcpu structure.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
When set_user_memory_region was moved to hypervisor crate, it was turned
into a safe function that wrapped around an unsafe call. All but one
call site had the safety statements removed. But safety statement was
not moved inside the wrapper function.
Add the safety statement back to help reasoning in the future. Also
remove that one last instance where the safety statement is not needed .
No functional change.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
That removes one more KVM-ism in VMM crate.
Note that there are more KVM specific code in those files to be split
out, but we're not at that stage yet.
No functional change.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Add a simple test to check that the data from the counters matches what
is expected and that the value of the counters increases after an
operation that will hit all counters.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Collate the virtio device counters in DeviceManager for each device that
exposes any and expose it through the recently added HTTP API.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The counters are a hash of device name to hash of counter name to u64
value. Currently the API is only implemented with a stub that returns an
empty set of counters.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Through the counters() function on the trait expose the accumulated
counters.
TEST=Observe that the counters from the VM match those from the tap on
the host (RX-TX inverted) and inside the guest (non inverted.)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The counters are a hash of counter name to (wrapping) u64 value. The
interpretation layer is responsible for converting this data into a
rate.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This corresponds to QEMU's 63659fe74e76f5c52854 commit.
the setattr code uses both fchmod and fchmodat in different cases,
however we only had fchmodat in the whitelist.
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
The structure is tightly coupled with KVM. It uses KVM specific
structures and calls. Add Kvm prefix to it.
Microsoft hypervisor will implement its own interrupt group(s) later.
No functional change intended.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Now that the VMM uses KVM_KVMCLOCK_CTRL from the KVM API, it must be
added to the seccomp filters list.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Through the newly added API notify_guest_clock_paused(), this patch
improves the vCPU pause operation by letting the guest know that each
vCPU is being paused. This is important to avoid soft lockups detection
from the guest that could happen because the VM has been paused for more
than 20 seconds.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
It's important that on restore path, the CpuManager's vCPU gets filled
with each new vCPU that is being created. In order to cover both boot
and restore paths, the list is being filled from the common function
create_vcpu().
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that the VMM uses both KVM_GET_CLOCK and KVM_SET_CLOCK from the KVM
API, they must be added to the seccomp filters list.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to maintain correct time when doing pause/resume and
snapshot/restore operations, this patch stores the clock value
on pause, and restore it on resume. Because snapshot/restore
expects a VM to be paused before the snapshot and paused after
the restore, this covers the migration use case too.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
When a request is made to increase the number of vCPUs in the VM attempt
to reuse any previously removed (and hence inactive) vCPUs before
creating new ones.
This ensures that the APIC ID is not reused for a different KVM vCPU
(which is not allowed) and that the APIC IDs are also sequential.
The two key changes to support this are:
* Clearing the "kill" bit on the old vCPU state so that it does not
immediately exit upon thread recreation.
* Using the length of the vcpus vector (the number of allocated vcpus)
rather than the number of active vCPUs (.present_vcpus()) to determine
how many should be created.
This change also introduced some new info!() debugging on the vCPU
creation/removal path to aid further development in the future.
TEST=Expanded test_cpu_hotplug test.
Fixes: #1338
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
After the vCPU has been ejected and the thread shutdown it is useful to
clear the "kill" flag so that if the vCPU is reused it does not
immediately exit upon thread recreation.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
These messages are intended to be useful to support debugging related to
vCPU hotplug/unplug issues.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Now that snapshot/restore is symmetrical, that is the VM must be paused
before it is snapshot and it must be resumed after it's been restored,
the integration test is updated accordingly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The same way the VM and the vCPUs are restored in a paused state, all
devices associated with the device manager must be restored in the same
paused state.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Because we need to pause the VM before it is snapshot, it should be
restored in a paused state to keep the sequence symmetrical. That's the
reason why the state machine regarding the valid VM's state transition
needed to be updated accordingly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
To follow a symmetrical model, and avoid potential race conditions, it's
important to restore a previously snapshot VM in a "paused" state.
The snapshot operation being valid only if the VM has been previously
paused.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
When the hypervisor crate was introduced, a few places that handled
errors were commented out in favor of unwrap, but that's bad practice.
Restore proper error handling in those places in this patch.
We cannot use from_raw_os_error anymore because it is wrapped deep under
hypervisor crate. Create new custom errors instead.
Fixes: e4dee57e81 ("arch, pci, vmm: Initial switch to the hypervisor crate")
Signed-off-by: Wei Liu <liuwe@microsoft.com>
This commit fixes some warnings introduced in the previous
hyperviosr crate PR.Removed some unused variables from arch/aarch64
module.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Start moving the vmm, arch and pci crates to being hypervisor agnostic
by using the hypervisor trait and abstractions. This is not a complete
switch and there are still some remaining KVM dependencies.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
Implement the vCPU state getter and setter separately from the initial
KVM Hypervisor trait implementation, mostly for readability purposes.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
For each of the traits we are defining kvm related structures
and add the trait implementation to the structs. For more information
please see the kvm-ioctls and kvm-bindings crate.
This is a standalone implementation that does not include the switch of
the Cloud-Hypervisor vmm and arch crates to it.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
As the only hypervisor that Cloud-Hypervisor supports is KVM, the
Hypervisor trait accomodates for the upcoming KVM implementation.
This trait will be instanciated at build time through hypervisor
specific features, i.e. it's not aiming at run-time selection of
hypervisors for Cloud-Hypervisor.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
This VM trait should be implemented by each underlying hypervisor.
Previously created hypervisor object should create the VM based on
already selected hypervisor. This is just the trait definition. For each
of supported hypervisor we need to implement the trait. Later we will
implement this trait for KVM and then Microsoft Hyper-V.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
This Vcpu trait should be implemented by each underlying hypervisor.
Previously created hypervisor object should create the VM based on
already selected hypervisor and Vm object should create this vcpu
object based on same hyperviosr. Each of this object should be
referenced by trait object i.e <dyn Vcpu>.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
The purpose of this trait is to add support for other hypervisors than
KVM, like e.g. Microsoft Hyper-V.
Further commits will define additional hypervisor related traits like
Vcpu and Vm. Each of the supported hypervisor will need to implement all
traits defined from the hypervisor crate.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Add multiple integration tests with various different CPU topologies and
check that they work as expected.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
There are two CPUID leaves for handling CPU topology, 0xb and 0x1f. The
difference between the two is that the 0x1f leaf (Extended Topology
Leaf) supports exposing multiple die packages.
Fixes: #1284
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The extended topology leaf (0x1f) also needs to have the APIC ID (which
is the KVM cpu ID) set. This mirrors the APIC ID set on the 0xb topology
leaf
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Rather than saving the individual parts into the CpuManager save the
full struct as it now also contains the topology data.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This needed to be updated to include specifying the boot and maxmium
vCPUs as well as the newly added topology for those vCPUs.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This allows the user to optionally specify the desired CPU topology. All
parts of the topology must be specified and the product of all parts
must match the maximum vCPUs.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This patch has been cherry-picked from the Firecracker tree. The
reference commit is 78ca0a942f32140465c67ea4b45d68c52c72d751.
Signed-off-by: Gabriel Ionescu <gbi@amazon.com>
Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
This patch has been cherry-picked from the Firecracker tree. The
reference commit is 6dbe8e021a64ba3742081741a7538cdfd93a102e.
Signed-off-by: Gabriel Ionescu <gbi@amazon.com>
Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
This patch has been cherry-picked from the Firecracker tree. The
reference commit is 78819f35f63f5777a58e3e1e774b3270b32881ed.
The vsock TX buffer flush operation would report inconsistent results,
under specific circumstances.
The flush operation is performed in two steps, since it's dealing with a
ring buffer, an the data to be flushed may wrap around. If the first
step was successful, but the second one failed, the whole flush
operation would report an error, thus causing flow control accounting to
lose track of the bytes that were successfully written by the first
pass.
This commit changes the flush behavior to always report success when
some data has been written to the backing stream.
Signed-off-by: Dan Horobeanu <dhr@amazon.com>
Signed-off-by: Gabriel Ionescu <gbi@amazon.com>
Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
This patch has been cherry-picked from the Firecracker tree. The
reference commit is 2da612a9cdce85c91fb54ab22d950ec6ccc93b27.
Fixed a bug introduced by a271d08f0b1ba0ee82761cd49244b6a8017bcede,
whereby the flow control accouting would be off by a few bytes, for
host-initiated connections.
The connection ack message ("OK <port_num><CR>") was accounted for as
data sent by the guest, so its length was substracted from the total
amount of data the guest was allowed to send.
This commit changes the way this ack message is sent, so that it
bypasses flow control accouting.
Signed-off-by: Dan Horobeanu <dhr@amazon.com>
Signed-off-by: Gabriel Ionescu <gbi@amazon.com>
Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
This patch has been cherry-picked from the Firecracker tree. The
reference commit is 109e631566350867dafa4b16c3919dfd1533eeea.
This commit changes the vsock connection state machine behavior to absorb
any EWOULDBLOCK errors recevied while handling an EPOLLOUT event. Previously,
this condition would lead to immediate connection termination.
Signed-off-by: Dan Horobeanu <dhr@amazon.com>
Signed-off-by: Gabriel Ionescu <gbi@amazon.com>
Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
This patch has been cherry-picked from the Firecracker tree. The
reference commit is 660d18cf7fee5b38c3b1b17a5da6544b9025909d.
Apparently, epoll_wait sometimes yields false EPOLLIN events (i.e. events
follwing which read() would fail with EWOULDBLOCK). This would cause the
vsock connection state machine to terminate connections, since an error
was detected on the underlying Unix socket.
This commit changes the vsock connection state machine code to handle such
erroneous EPOLLIN events by absorbing EWOULDBLOCK read() errors.
Signed-off-by: Dan Horobeanu <dhr@amazon.com>
Signed-off-by: Gabriel Ionescu <gbi@amazon.com>
Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
This patch has been cherry-picked from the Firecracker tree. The
reference commit is 1cc8b8a678eb28b20f5843556bdb7fbb2dfa6284.
Fixed a logical error in the vsock flow control, that would cause credit
update packets to not be sent at the right time.
Signed-off-by: Dan Horobeanu <dhr@amazon.com>
Signed-off-by: Gabriel Ionescu <gbi@amazon.com>
Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
This patch has been cherry-picked from the Firecracker tree. The
reference commit is d2475773557c82d2abad2fc8bdf69e7d01444109.
Fixed a vsock muxer issue that would cause a connection to be removed
from the RX queue, even though it still had pending RX data.
Signed-off-by: Dan Horobeanu <dhr@amazon.com>
Signed-off-by: Gabriel Ionescu <gbi@amazon.com>
Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
This patch adds `is_empty` method to VsockPacket to fix the
following clippy error:
error: item `vsock::packet::VsockPacket` has a public `len` method but no corresponding `is_empty` method
--> vm-virtio/src/vsock/packet.rs:100:1
|
100 | / impl VsockPacket {
101 | | /// Create the packet wrapper from a TX virtq chain head.
102 | | ///
103 | | /// The chain head is expected to hold valid packet header data. A following packet buffer
... |
334 | | }
335 | | }
| |_^
|
= note: `-D clippy::len-without-is-empty` implied by `-D warnings`
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#len_without_is_empty
Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
Currently, not every feature of the cloud-hypervisor is enabled
on AArch64, which means that on AArch64 machines, the
`run_unit_tests.sh` needs to be tailored and some unit test cases
should be run on x86_64 only.
Also this commit fixes the typo and unifies `Arm64` and `AArch64`
in the AArch64 document.
Signed-off-by: Henry Wang <Henry.Wang@arm.com>
Its test case calls remove unconditionally. Instead of making the test
code call remove conditionally, removing the pci_support dependency
simplifies things -- that function is just a wrapper around HashMap's
remove function anyway.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
cloud-hypervisor: 763.978581807s: ERROR:pci/src/vfio.rs:651 -- failed to remove all guest memory regions from iommu table
when poweroff a vm with vfio device, clh will finally remove all guest memory region from iommu table
with the method unset_dma_map, not method setup_dma_map.
Signed-off-by: LiYa'nan <oliverliyn@gmail.com>
Upon PCI hotplug, the VMM now returns some information about the device
name and the associated b/d/f. This patch extends the integration tests
so that we validate the response is the one that is expected.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now that PCI device hotplug returns a response, the OpenAPI definition
must reflect it, describing what is expected to be received.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to provide a more comprehensive b/d/f to the user, the
serialization of PciDeviceInfo is implemented manually to control the
formatting.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This patch completes the series by connecting the dots between the HTTP
frontend and the device manager backend.
Any request to hotplug a VFIO, disk, fs, pmem, net, or vsock device will
now return a response including the device name and the place of the
device in the PCI topology.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Pass from the device manager to the calling code the information about
the PCI device that has just been hotplugged.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to provide the device name and PCI b/d/f associated with a
freshly hotplugged device, the hotplugging functions from the device
manager return a new structure called PciDeviceInfo.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
- [Firecracker and crosvm](#firecracker-and-crosvm)
- [5. Community](#5-community)
- [Contribute](#contribute)
- [Join us](#join-us)
# 1. What is Cloud Hypervisor?
**This project is an experiment and should not be used with production workloads.**
Cloud Hypervisor is an open source Virtual Machine Monitor (VMM) that runs on top of [KVM](https://www.kernel.org/doc/Documentation/virtual/kvm/api.txt).
The project focuses on exclusively running modern, cloud workloads, on top of a limited set of hardware architectures and platforms.
Cloud workloads refers to those that are usually run by customers inside a cloud provider. For our purposes this means modern
Linux* distributions with most I/O handled by paravirtualised devices (i.e. virtio), no requirement for legacy devices and recent CPUs and KVM.
Cloud workloads refers to those that are usually run by customers inside a cloud provider. For our purposes this means modern operating systems with most I/O handled by paravirtualised devices (i.e. virtio), no requirement for legacy devices, and 64-bit CPUs.
Cloud Hypervisor is implemented in [Rust](https://www.rust-lang.org/) and is based on the [rust-vmm](https://github.com/rust-vmm) crates.
@@ -37,27 +36,24 @@ Cloud Hypervisor is implemented in [Rust](https://www.rust-lang.org/) and is bas
### High Level
* KVM and KVM only based
* Minimal emulation
* Low latency
* Low memory footprint
* Low complexity
* High performance
* Small attack surface
* 64-bit support only
* Build time configurable CPU, memory, PCI and NVDIMM hotplug
* Machine to machine migration
- KVM based
- Minimal emulation
- Low latency
- Low memory footprint
- Low complexity
- High performance
- Small attack surface
- 64-bit support only
- CPU, memory, PCI hotplug
- Machine to machine migration
### Architectures
`cloud-hypervisor` only supports the `x86-64`CPU architecture for now.
We're planning to add support for the `AArch64` architecture in the future.
Cloud Hypervisor supports the `x86-64`and `AArch64` architectures. There are some small differences in functionality between the two architectures (see [#1125](https://github.com/cloud-hypervisor/cloud-hypervisor/issues/1125)).
### Guest OS
*`64-bit Linux`
Support for *modern* 64-bit Windows guest is being evaluated.
Cloud Hypervisor supports `64-bit Linux` with support for _modern_ 64-bit Windows guests currently under development.
@@ -156,8 +171,7 @@ Multiple arguments can be given to the `--disk` parameter.
#### Building your kernel
`cloud-hypervisor` also supports direct kernel boot into a `vmlinux` ELF kernel
image. In order to support virtio-fs and virtio-iommu we have our own development branch. You are of course able to use your own kernel but these instructions will continue with the version that we develop and test against.
Cloud Hypervisor also supports direct kernel boot into a `vmlinux` ELF kernel or `bzImage`. In order to support virtio-fs and virtio-iommu we have our own development branch. You are of course able to use your own kernel but these instructions will continue with the version that we develop and test against.
Direct kernel boot to userspace should work with a rootfs from most distributions.
## Hot Plug
This [document](https://github.com/cloud-hypervisor/cloud-hypervisor/blob/master/docs/hotplug.md) details how to add devices to
a running VM. Currently only CPU hot plug is supported.
Cloud Hypervisor supports hotplug of CPUs, passthrough devices (VFIO), `virtio-{net,block,pmem,fs,vsock}` and memory resizing. This [document](https://github.com/cloud-hypervisor/cloud-hypervisor/blob/master/docs/hotplug.md) details how to add devices to
a running VM.
## Device Model
Follow this [documentation](https://github.com/cloud-hypervisor/cloud-hypervisor/blob/master/docs/device_model.md).
Details of the device model can be found in this [documentation](https://github.com/cloud-hypervisor/cloud-hypervisor/blob/master/docs/device_model.md).
## TODO
We are not tracking the `cloud-hypervisor` TODO list from a specific git tracked file but through
We are not tracking the Cloud Hypervisor TODO list from a specific git tracked file but through
In order to satisfy the design goal of having a high-performance, security-focused hypervisor the decision
was made to use the [Rust](https://www.rust-lang.org/) programming language.
The language's strong focus on memory and thread safety makes it an ideal candidate for implementing VMMs
The language's strong focus on memory and thread safety makes it an ideal candidate for implementing VMMs.
Instead of implementing the VMM components from scratch, `cloud-hypervisor` is importing the [rust-vmm](https://github.com/rust-vmm)
Instead of implementing the VMM components from scratch, Cloud Hypervisor is importing the [rust-vmm](https://github.com/rust-vmm)
crates, and sharing code and architecture together with other VMMs like e.g. Amazon's [Firecracker](https://firecracker-microvm.github.io/)
and Google's [crosvm](https://chromium.googlesource.com/chromiumos/platform/crosvm/).
`cloud-hypervisor` embraces the rust-vmm project goals, which is to be able to share and re-use
as many virtualization crates as possible. As such, the `cloud-hypervisor` relationship with the rust-vmm
Cloud Hypervisor embraces the rust-vmm project goals, which is to be able to share and re-use
as many virtualization crates as possible. As such, the Cloud Hypervisor relationship with the rust-vmm
project is twofold:
1. It will use as much of the rust-vmm code as possible. Any new rust-vmm crate that's relevant to the project
goals will be integrated as soon as possible.
2. As it is likely that the rust-vmm project will lack some of the features that `cloud-hypervisor` needs (e.g. ACPI,
VFIO, vhost-user, etc), we will be using the `cloud-hypervisor` VMM to implement and test them, and contribute them
2. As it is likely that the rust-vmm project will lack some of the features that Cloud Hypervisor needs (e.g. ACPI,
VFIO, vhost-user, etc), we will be using the Cloud Hypervisor VMM to implement and test them, and contribute them
back to the rust-vmm project.
## Firecracker and crosvm
A large part of the `cloud-hypervisor` code is based on either the Firecracker or the crosvm projects implementations.
A large part of the Cloud Hypervisor code is based on either the Firecracker or the crosvm projects implementations.
Both of these are VMMs written in Rust with a focus on safety and security, like Cloud Hypervisor.
However we want to emphasize that the Cloud Hypervisor project is neither a fork nor a reimplementation of any of those
@@ -287,6 +301,11 @@ crates as possible.
# 5. Community
The Cloud Hypervisor project follows the governance, and community guidelines described in
the [Community](https://github.com/cloud-hypervisor/community) repository.
## Contribute
We are working on building a global, diverse and collaborative community around the Cloud Hypervisor project.
Anyone who is interested in [contributing](CONTRIBUTING.md) to the project is welcome to participate.
@@ -298,27 +317,3 @@ etc, are all equal and welcome means of contribution. See the [CONTRIBUTING](CON
Get an [invite to our Slack channel](https://join.slack.com/t/cloud-hypervisor/shared_invite/enQtNjY3MTE3MDkwNDQ4LWQ1MTA1ZDVmODkwMWQ1MTRhYzk4ZGNlN2UwNTI3ZmFlODU0OTcwOWZjMTkwZDExYWE3YjFmNzgzY2FmNDAyMjI)
and [join us on Slack](https://cloud-hypervisor.slack.com/).
# 6. Security
**Reporting a Potential Security Vulnerability**: If you have discovered
potential security vulnerability in this project, please send an e-mail to
secure@intel.com. For issues related to Intel Products, please visit
https://security-center.intel.com.
It is important to include the following details:
- The projects and versions affected
- Detailed description of the vulnerability
- Information on known exploits
Vulnerability information is extremely sensitive. Please encrypt all security
vulnerability reports using our *PGP key*
A member of the Intel Product Security Team will review your e-mail and
contact you to to collaborate on resolving the issue. For more information on
how Intel works to resolve security issues, see: *Vulnerability Handling
Before building, a hack trick need to be performed to get rid of some build error in vmm component. See [this](https://github.com/cloud-hypervisor/kvm-bindings/pull/1) for more info about this temporary workaround.
Using PCI devices requires GICv3-ITS for MSI messaging. GICv3-ITS is very common in modern servers.
```bash
sed -i 's/"with-serde",\ //g' vmm/Cargo.toml
```
The support of AArch64 is in very early stage, only Virtio devices with MMIO tranport is available.
@@ -52,14 +52,14 @@ Direct kernel boot option is preferred since we need to provide the custom kerne
Because _vhost-user_ expects a dedicated process (__virtiofsd__ in this case) to be able to access the guest RAM to communicate through the _virtqueues_ with the driver running in the guest, `--memory` option needs to be slightly modified. It needs to specify a backing file for the memory so that an external process can access it.
Assuming you have `clear-kvm.img` and `custom-vmlinux.bin` on your system, here is the __cloud-hypervisor__ command you need to run:
Assuming you have `focal-server-cloudimg-amd64.raw` and `custom-vmlinux.bin` on your system, here is the __cloud-hypervisor__ command you need to run:
@@ -5,9 +5,7 @@ Currently Cloud Hypervisor only support hot plugging of CPU devices.
## Kernel support
For hotplug on Cloud Hypervisor ACPI GED support is needed. This can either be achieved by turning on `CONFIG_ACPI_REDUCED_HARDWARE_ONLY`
or by using this kernel patch (available in 5.5rc1 and later): https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/patch/drivers/acpi/Makefile?id=ac36d37e943635fc072e9d4f47e40a48fbcdb3f0
This patch is integrated into the Clear Linux KVM and cloudguest images.
or by using this kernel patch (available in 5.5-rc1 and later): https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/patch/drivers/acpi/Makefile?id=ac36d37e943635fc072e9d4f47e40a48fbcdb3f0
## CPU Hot Plug
@@ -22,13 +20,14 @@ To use CPU hotplug start the VM with the number of max vCPUs greater than the nu
The extra vCPU threads will be created and advertised to the running kernel. The kernel does not bring up the CPUs immediately and instead the user must "online" them from inside the VM:
@@ -57,31 +56,36 @@ After a reboot the added CPUs will remain.
Removing CPUs works similarly by reducing the number in the "desired_vcpus" field of the reisze API. The CPUs will be automatically offlined inside the guest so there is no need to run any commands inside the guest:
As per adding CPUs to the guest, after a reboot the VM will be running with the reduced number of vCPUs.
## Memory Hot Plug
Extra memory can be added from a runing Cloud Hypervisor instance. This is controlled by two mechanisms:
### ACPI method
Extra memory can be added from a running Cloud Hypervisor instance. This is controlled by two mechanisms:
1. Allocating some of the guest physical address space for hotplug memory.
2. Making a HTTP API request to the VMM to ask for a new amount of RAM to be assigned to the VM. In the case of expanding the memory for the VM the new memory will be hotplugged into the running VM, if reducing the size of the memory then change will take effect after the next reboot.
To use memory hotplug start the VM specifying some size RAM in the "hotplug_size" parameter to the memory configuration. Not all the memory specified in this parameter will be available to hotplug as there are spacing and alignment requirements so it is recommended to make it larger than the hotplug RAM needed.
To use memory hotplug start the VM specifying some size RAM in the `hotplug_size` parameter to the memory configuration. Not all the memory specified in this parameter will be available to hotplug as there are spacing and alignment requirements so it is recommended to make it larger than the hotplug RAM needed.
Because the ACPI method is the default, there is no need to add the extra option `hotplug_method=acpi`.
The new memory is now available to use inside the VM:
@@ -110,4 +114,135 @@ Due to guest OS limitations is is necessary to ensure that amount of memory adde
The same API can also be used to reduce the desired RAM for a VM but the change will not be applied until the VM is rebooted.
Memory and CPU resizing can be combined together into the same HTTP API request.
Memory and CPU resizing can be combined together into the same HTTP API request.
### virtio-mem method
Extra memory can be added and removed from a running Cloud Hypervisor instance. This is controlled by two mechanisms:
1. Allocating some of the guest physical address space for hotplug memory.
2. Making a HTTP API request to the VMM to ask for a new amount of RAM to be assigned to the VM.
To use memory hotplug start the VM specifying some size RAM in the `hotplug_size` parameter along with `hotplug_method=virtio-mem` to the memory configuration.
The new memory is now available to use inside the VM:
```shell
free -h
total used free shared buff/cache available
Mem: 3.0Gi 71Mi 2.8Gi 0.0Ki 47Mi 2.8Gi
Swap: 32Mi 0B 32Mi
```
The same API can also be used to reduce the desired RAM for a VM. It is important to note that reducing RAM size might only partially work, as the guest might be using some of it.
## PCI Device Hot Plug
Extra PCI devices can be added and removed from a running Cloud Hypervisor instance. This is controlled by making a HTTP API request to the VMM to ask for the additional device to be added, or for the existing device to be removed.
To use PCI device hotplug start the VM with the HTTP server.
00:01.0 Unassigned class [ffff]: Red Hat, Inc. Virtio console (rev 01)
00:02.0 Mass storage controller: Red Hat, Inc. Virtio block device (rev 01)
00:03.0 Unassigned class [ffff]: Red Hat, Inc. Virtio RNG (rev 01)
```
After a reboot the added PCI device will remain.
### Remove PCI device
Removing a PCI device works the same way for all kind of PCI devices. The unique identifier related to the device must be provided. This identifier can be provided by the user when adding the new device, or by default Cloud Hypervisor will assign one.
* Developers who want to understand what log level to use and when,
* Users who want to debug issues with running their workloads in Cloud Hypervisor
## Control
The number of `-v` parameters passed to the `cloud-hypervisor` binary will determine the log level. Currenly the default is log messages up to `WARN:` (`warn!`) are included by default. The `--log-file` allows the log to be sent to a location other than `stderr`.
## Levels
### `error!()`
For immediate, unrecoverable errors where it does not make sense for the execution to continue as the behaviour of the VM is considerablely impacted.
Cloud Hypervisor should exit shortly after reporting this error (with a non-zero exit code). Generally this should be used during initial construction of the VM state before the virtual CPUs have begun running code.
A typical situation where this might occur is when the user is using command line options that conflict with each other or is trying to use a file that is not present on the filesystem.
Users should react to this error by checking their initial VM configuration.
### `warn!()`
A serious problem has occured but the execution of the VM can continue although some functionality might be impacted.
A typical example of where this level of message should be generated is during an API call request that cannot be fulfilled.
The user should investigate the meaning of this warning and take steps to ensure the correct functionality.
### `info!()`
Use `-v` to enable.
This level is for the benefit of developers. It should be used for sporadic and infrequent messages. The same message should not "spam" the logs. The VM should be usable when this level of debugging is enabled and trying to use `stdin/stdout` and the logs are going to `stderr`.
### `debug!()`
Use `-vv` to enable.
For the most verbose of logging messages. It is acceptable to "spam" the log with repeated invocations of the same message. This level of logging would be combined with `--log-file`.
Cloud Hypervisor supports using a MACVTAP device which is derived from a MACVLAN. Full details of configuring MACVLAN or MACVTAP is out of scope of this document. However the example below indicates how to bridge the guest directly onto the the network the host is on. Due to the lack of hairpin mode it not usually possible to reach the guest directly from the host.
```bash
# The MAC address must be attached to the macvtap and be used inside the guest
mac="c2:67:4f:53:29:cb"
# Host network adapter to bridge the guest onto
host_net="eno1"
# Create the macvtap0 as a new virtual MAC associated with the host network
sudo ip link add link "$host_net" name macvtap0 type macvtap
sudo ip link set macvtap0 address "$mac" up
sudo ip link show macvtap0
# A new character device is created for this interface
tapindex=$(< /sys/class/net/macvtap0/ifindex)
tapdevice="/dev/tap$tapindex"
# Ensure that we can access this device
sudo chown "$UID.$UID""$tapdevice"
# Use --net fd=3 to point to fd 3 which the shell has opened to point to the /dev/tapN device
target/debug/cloud-hypervisor \
--kernel ~/src/linux/vmlinux \
--disk path=~/workloads/focal.raw \
--cpus boot=1 --memory size=512M \
--cmdline "root=/dev/vda1 console=hvc0"\
--net fd=3,mac=$mac 3<>$"$tapdevice"
```
As the guest is now connected to the same L2 network as the host you can obtain an IP address based on your host network (potentially including via DHCP)
This option is mandatory when using the `--memory` parameter.
Value is an unsigned integer of 64 bits.
_Example_
```
--memory size=1G
```
### `mergeable`
Specifies if the pages from the guest RAM must be marked as _mergeable_. In
case this option is `true` or `on`, the pages will be marked with `madvise(2)`
to let the host kernel know which pages are eligible for being merged by the
KSM daemon.
This option can be used when trying to reach a higher density of VMs running
on a single host, as it will reduce the amount of memory consumed by each VM.
By default this option is turned off.
_Example_
```
--memory size=1G,mergeable=on
```
### `shared`
Specifies if the memory must be `mmap(2)` with `MAP_SHARED` flag.
By sharing a memory mapping, one can share the guest RAM with other processes
running on the host. One can use this option when running vhost-user devices
as part of the VM device model, as they will be driven by standalone daemons
needing access to the guest RAM content.
By default this option is turned off, which results in performing `mmap(2)`
with `MAP_PRIVATE` flag.
_Example_
```
--memory size=1G,shared=on
```
### `hugepages` and `hugepage_size`
Specifies if the memory must be created and `mmap(2)` with `MAP_HUGETLB` and size
flags. This performs a memory mapping relying on the specified huge page size. If no huge page size is supplied the system's default huge page size is used.
By using hugepages, one can improve the overall performance of the VM, assuming
the guest will allocate hugepages as well. Another interesting use case is VFIO
as it speeds up the VM's boot time since the amount of IOMMU mappings are
reduced.
The user is responsible for ensuring there are sufficient huge pages of the specified size for the VMM to use. Failure to do so may result in strange VMM behaviour.
By default this option is turned off.
_Example_
```
--memory size=1G,hugepages=on,hugepage_size=2M
```
### `hotplug_method`
Selects the way of adding and/or removing memory to/from a booted VM.
Possible values are `acpi` and `virtio-mem`. Default value is `acpi`.
_Example_
```
--memory size=1G,hotplug_method=acpi
```
### `hotplug_size`
Amount of memory that can be dynamically added to the VM.
Value is an unsigned integer of 64 bits. A value of 0 is invalid.
_Example_
```
--memory size=1G,hotplug_size=1G
```
### `hotplugged_size`
Amount of memory that will be dynamically added to the VM at boot. This option
allows for starting a VM with a certain amount of memory that can be reduced
during runtime.
This is only valid when the `hotplug_method` is `virtio-mem` as it does not
make sense for the `acpi` use case. When using ACPI, the memory can't be
resized after it has been extended.
This option is only valid when `hotplug_size` is specified, and its value can't
exceed the value of `hotplug_size`.
Value is an unsigned integer of 64 bits. A value of 0 is invalid.
`MemoryZoneConfig` or what is known as `--memory-zone` from the CLI perspective
is a power user parameter. It allows for a full description of the guest RAM,
describing how every memory region is backed and exposed to the guest.
```rust
structMemoryZoneConfig{
id: String,
size: u64,
file: Option<PathBuf>,
shared: bool,
hugepages: bool,
host_numa_node: Option<u32>,
hotplug_size: Option<u64>,
hotplugged_size: Option<u64>,
}
```
```
--memory-zone <memory-zone> User defined memory zone parameters "size=<guest_memory_region_size>,file=<backing_file>,shared=on|off,hugepages=on|off,host_numa_node=<node_id>,id=<zone_identifier>,hotplug_size=<hotpluggable_memory_size>,hotplugged_size=<hotplugged_memory_size>"
```
This parameter expects one or more occurences, allowing for a list of memory
zones to be defined. It must be used with `--memory size=0`, clearly indicating
that the memory will be described through advanced parameters.
Each zone is given a list of options which we detail through the following
sections.
### `id`
Memory zone identifier. This identifier must be unique, otherwise an error will
be returned.
This option is useful when referring to a memory zone previously created. In
particular, the `--numa` parameter can associate a memory zone to a specific
NUMA node based on the memory zone identifier.
This option is mandatory when using the `--memory-zone` parameter.
Value is a string.
_Example_
```
--memory size=0
--memory-zone id=mem0,size=1G
```
### `size`
Size of the memory zone.
This option is mandatory when using the `--memory-zone` parameter.
Value is an unsigned integer of 64 bits.
_Example_
```
--memory size=0
--memory-zone id=mem0,size=1G
```
### `file`
Path to the file backing the memory zone. This can be either a file or a
directory. In case of a file, it will be opened and used as the backing file
for the `mmap(2)` operation. In case of a directory, a temporary file with no
hard link on the filesystem will be created. This file will be used as the
backing file for the `mmap(2)` operation.
This option can be particularly useful when trying to back a part of the guest
RAM with a well known file. In the context of the snapshot/restore feature, and
if the provided path is a file, the snapshot operation will not perform any
copy of the guest RAM content for this specific memory zone since the user has
access to it and it would duplicate data already stored on the current
filesystem.
Value is a string.
_Example_
```
--memory size=0
--memory-zone id=mem0,size=1G,file=/foo/bar
```
### `shared`
Specifies if the memory zone must be `mmap(2)` with `MAP_SHARED` flag.
By sharing a memory zone mapping, one can share part of the guest RAM with
other processes running on the host. One can use this option when running
vhost-user devices as part of the VM device model, as they will be driven
by standalone daemons needing access to the guest RAM content.
By default this option is turned off, which result in performing `mmap(2)`
with `MAP_PRIVATE` flag.
_Example_
```
--memory size=0
--memory-zone id=mem0,size=1G,shared=on
```
### `hugepages`
Specifies if the memory zone must be `mmap(2)` with `MAP_HUGETLB` and
`MAP_HUGE_2MB` flags. This performs a memory zone mapping relying on 2MiB
pages instead of the default 4kiB pages.
By using hugepages, one can improve the overall performance of the VM, assuming
the guest will allocate hugepages as well. Another interesting use case is VFIO
as it speeds up the VM's boot time since the amount of IOMMU mappings are
reduced.
By default this option is turned off.
_Example_
```
--memory size=0
--memory-zone id=mem0,size=1G,hugepages=on
```
### `host_numa_node`
Node identifier of a node present on the host. This option will let the user
pick a specific NUMA node from which the memory must be allocated. After the
memory zone is `mmap(2)`, the NUMA policy for this memory mapping will be
applied through `mbind(2)`, relying on the provided node identifier. If the
node does not exist on the host, the call to `mbind(2)` will fail.
This option is useful when trying to back a VM memory with a specific type of
memory from the host. Assuming a host has two types of memory, with one slower
than the other, each related to a distinct NUMA node, one could create a VM
with slower memory accesses by backing the entire guest RAM from the furthest
NUMA node on the host.
This option also gives the opportunity to create a VM with non uniform memory
accesses as one could define a first memory zone backed by fast memory, and a
second memory zone backed by slow memory.
Value is an unsigned integer of 32 bits.
_Example_
```
--memory size=0
--memory-zone id=mem0,size=1G,host_numa_node=0
```
### `hotplug_size`
Amount of memory that can be dynamically added to the memory zone. Since
`virtio-mem` is the only way of resizing a memory zone, one must specify
the `hotplug_method=virtio-mem` to the `--memory` parameter.
Value is an unsigned integer of 64 bits. A value of 0 is invalid.
_Example_
```
--memory size=0,hotplug_method=virtio-mem
--memory-zone id=mem0,size=1G,hotplug_size=1G
```
### `hotplugged_size`
Amount of memory that will be dynamically added to a memory zone at VM's boot.
This option allows for starting a VM with a certain amount of memory that can
be reduced during runtime.
This is only valid when the `hotplug_method` is `virtio-mem` as it does not
make sense for the `acpi` use case. When using ACPI, the memory can't be
resized after it has been extended.
This option is only valid when `hotplug_size` is specified, and its value can't
exceed the value of `hotplug_size`.
Value is an unsigned integer of 64 bits. A value of 0 is invalid.
`NumaConfig` or what is known as `--numa` from the CLI perspective has been
introduced to define a guest NUMA topology. It allows for a fine description
about the CPUs and memory ranges associated with each NUMA node. Additionally
it allows for specifying the distance between each NUMA node.
```rust
structNumaConfig{
id: u32,
cpus: Option<Vec<u8>>,
distances: Option<Vec<NumaDistance>>,
memory_zones: Option<Vec<String>>,
}
```
```
--numa <numa> Settings related to a given NUMA node "id=<node_id>,cpus=<cpus_id>,distances=<list_of_distances_to_destination_nodes>,memory_zones=<list_of_memory_zones>"
```
### `guest_numa_id`
Node identifier of a guest NUMA node. This identifier must be unique, otherwise
an error will be returned.
This option is mandatory when using the `--numa` parameter.
Value is an unsigned integer of 32 bits.
_Example_
```
--numa guest_numa_id=0
```
### `cpus`
List of virtual CPUs attached to the guest NUMA node identified by the
`guest_numa_id` option. This allows for describing a list of CPUs which
must be seen by the guest as belonging to the NUMA node `guest_numa_id`.
One can use this option for a fine grained description of the NUMA topology
regarding the CPUs associated with it, which might help the guest run more
efficiently.
Multiple values can be provided to define the list. Each value is an unsigned
integer of 8 bits.
For instance, if one needs to attach all CPUs from 0 to 4 to a specific node,
the syntax using `-` will help define a contiguous range with `cpus=0-4`. The
same example could also be described with `cpus=0:1:2:3:4`.
A combination of both `-` and `:` separators is useful when one might need to
describe a list containing all CPUs from 0 to 99 and the CPU 255, as it could
simply be described with `cpus=0-99:255`.
_Example_
```
--cpus boot=8
--numa guest_numa_id=0,cpus=1-3:7
--numa guest_numa_id=1,cpus=0:4-6
```
### `distances`
List of distances between the current NUMA node referred by `guest_numa_id`
and the destination NUMA nodes listed along with distances. This option let
the user choose the distances between guest NUMA nodes. This is important to
provide an accurate description of the way non uniform memory accesses will
perform in the guest.
One or more tuple of two values must be provided through this option. The first
value is an unsigned integer of 32 bits as it represents the destination NUMA
node. The second value is an unsigned integer of 8 bits as it represents the
distance between the current NUMA node and the destination NUMA node. The two
values are separated by `@` (`value1@value2`), meaning the destination NUMA
node `value1` is located at a distance of `value2`. Each tuple is separated
from the others with `:` separator.
For instance, if one wants to define 3 NUMA nodes, with each node located at
different distances, it can be described with the following example.
_Example_
```
--numa guest_numa_id=0,distances=1@15:2@25
--numa guest_numa_id=1,distances=0@15:2@20
--numa guest_numa_id=2,distances=0@25:1@20
```
### `memory_zones`
List of memory zones attached to the guest NUMA node identified by the
`guest_numa_id` option. This allows for describing a list of memory ranges
which must be seen by the guest as belonging to the NUMA node `guest_numa_id`.
This option can be very useful and powerful when combined with `host_numa_node`
option from `--memory-zone` parameter as it allows for creating a VM with non
uniform memory accesses, and let the guest know about it. It allows for
exposing memory zones through different NUMA nodes, which can help the guest
workload run more efficiently.
Multiple values can be provided to define the list. Each value is a string
referring to an existing memory zone identifier. Values are separated from
cloud-hypervisor can emulate one or more virtual network interfaces, represented at the hypervisor host by [tap devices](https://www.kernel.org/doc/Documentation/networking/tuntap.txt). This guide briefly describes, in a manual and distribution neutral way, how to setup and use networking with cloud-hypevisor.
cloud-hypervisor can emulate one or more virtual network interfaces, represented at the hypervisor host by [tap devices](https://www.kernel.org/doc/Documentation/networking/tuntap.txt). This guide briefly describes, in a manual and distribution neutral way, how to setup and use networking with cloud-hypervisor.
## Multiple queue support for net devices ##
@@ -14,7 +14,7 @@ Note:
- Currently, it does not support to use ethtool to change the combined queue numbers in guest.
- Multiple queue is enabled for vhost-user-net backend in cloud-hypervisor, however, multiple thread is not added to handle mq, thus, the performance for vhost-user-net backend is not supposed to be improved. The multiple thread will be added for backend later.
- Performance test for vhost-user-net will be covered once vhost-user-net backend has mulitple thread supported.
- Performance test for vhost-user-net will be covered once vhost-user-net backend has multiple thread supported.
- Performance test for virtio-net is done by comparing 2 queue pairs with 1 queue pairs, that to run 2 iperf3 sessions in the same test environments, throughput is improved about 37%.
## Start cloud-hypervisor with net devices
@@ -25,9 +25,9 @@ Use one `--net` command-line argument from cloud-hypervisor to specify the emula
`perf` can be used to profile the `cloud-hypervisor` binary but it is necessary to make some modifications to the the build in order to produce a binary that gives useful results.
## Building a suitable binary
Modify the `Cargo.toml` file to add `debug = 1` to the `[profile.release]` block. It should look like this:
```
[profile.release]
lto = true
debug = 1
```
This adds the symbol information to the release binary but does not otherwise affect the performance.
The binary must also be built with frame pointers included so that the call graph can be captured by the profiler.
$ perf record -g target/release/cloud-hypervisor \
--kernel ~/src/linux/vmlinux \
--pmem file=~/workloads/focal.raw \
--cpus boot=1 --memory size=1G \
--cmdline "root=/dev/pmem0p1 console=ttyS0" \
--serial tty --console off \
--api-socket=/tmp/api1
```
For analysing the samples:
```
$ perf report -g
```
If profiling with a network device attached either the TAP device must be already created and configured or the profiling must be done as root so that the TAP device can be created.
Cloud Hypervisor supports UEFI boot through the utilization of the EDK II based UEFI firmware.
## Building UEFI Firmware
To avoid any unnecessary issues, it is recommended to use Ubuntu 18.04 and its default toolset. Any other compatible Linux distribution is otherwise suitable, however it is suggested to use a temporary Docker container with Ubuntu 18.04 for a quick build on an existing Linux machine.
The commands below will compile an OVMF firmware suitable for Cloud Hypervisor.
After the successful build, the resulting firmware binaries are available under `Build/OvmfCh/DEBUG_GCC5/FV` underneath the edk2 checkout.
## Using OVMF Binaries
Any UEFI capable image can be booted using the Cloud Hypervisor specific firmware. Windows guests under Cloud Hypervisor only support UEFI boot, therefore OVMF is mandatory there.
To make Cloud Hypervisor use UEFI boot, pass the `OVMF.fd` file path as an argument to the `--kernel` option. The firmware file will be opened in read only mode.
The same firmware can be used with Cloud Hypervisor or with QEMU. This is particularly useful if using QEMU for the preparation phase.
## Building UEFI Firmware with Compatibility Support Module (CSM)
CSM is a module that allows to boot legacy operating systems using the OVMF firmware. OVMF can embed a CSM build of SeaBIOS. To build the SeaBIOS with CSM support, add `CONFIG_CSM=y` to `.config` before the build. The outcome `out/Csm16.bin` is to be moved into `OvmfPkg/Csm/Csm16/Csm16.bin` before OVMF is built. Then, the OVMF build will have to be passed the `-D CSM_ENABLE` option in order to generate a legacy aware UEFI firmware. At the current stage, all the necessary patches are included in the Cloud Hypervisor specific [SeaBIOS branch](https://github.com/cloud-hypervisor/seabios/tree/ch). Taking into account the previous instructions, the modified command sequence to compile an OVMF binary with CSM support is the following one:
Please note, that the CSM support has currently only been tested with Linux guests. There are no plans to provide legacy support for other OSes (e.g. Windows).
Starting with the release version [0.10.0](https://github.com/cloud-hypervisor/cloud-hypervisor/releases/tag/v0.10.0), Cloud Hypervisor supports Windows guests.
__Requirements__
- Host with KVM enabled
- [UEFI](uefi.md) capable Windows guest image with Virtio drivers integrated
Any modern Windows Server version is compatible. Cloud Hypervisor has been successfully tested with Windows Server 2019 and Windows Server Core 2004.
At the current stage, only UEFI capable Windows images are supported. This implies the presence of the OVMF firmware during the Windows installation and in any subsequent usage. BIOS boot is not supported.
The subsequent sections will tell, in detail, how to prepare an appropriate Windows image.
## Image Preparation
### Installation using the stock Windows ISO
__Prerequisites__
- QEMU, version >=5.0.0 is recommended.
- Windows installation ISO. Obtained through MSDN, Visual Studio subscription, evaluation center, etc.
- With the suggested image size of 30G, there should be enough free disk space to hold the installation ISO and any other necessary files
This step currently requires QEMU to install Windows onto the guest. QEMU is only used at the preparation stage, the resulting image is then fully functional with Cloud Hypervisor.
Preparing several command parts as these will be used in the follow up sections as well.
Before the installation can proceed, point the Windows installation program to the VirtIO disk and install the necessary storage controller drivers. After that, the attached hard drive will become visible and the actual installation can commence.
After the installation has completed, proceed further to the configuration section. QEMU will be needed at least once more to enable the Windows Special Administration Console (SAC) and to possibly install extra device drivers.
## Image Usage
The basic command to boot a Windows image. The configuration section should be checked before executing it for the first time.
```shell
cloud-hypervisor \
--kernel ./$OVMF_DIR/OVMF.fd \
--disk path=./$IMG_FILE\
--cpus boot=1,kvm_hyperv=on \
--memory size=4G \
--serial tty \
--console off \
--net tap=
```
It is necessary to always:
- Carry the OVMF firmware in the `--kernel` option
- Add `kvm_hyperv=on` to the `--cpus` option
In cases where the host processor supports address space > 39 bits, it might be necessary to limit the address space. It can be done by appending the option `max_phys_bits=X` to the `--cpus` parameter, where `X` is the number of bits to be supported. Windows was tested to support at least 39-bit address space.
To daemonize the Cloud Hypervisor process, `nohup` can be used. Some STDIO redirections might need to be done. In a simple case it is sufficient to just redirect all the output to `/dev/null`.
## Image Configuration
### Device Drivers
After the Windows installation has finished under QEMU, there might be still devices with no drivers installed. This might happen for example, when a device was not used during the installation. In particular it is important to ensure that the VirtIO network device is setup correctly because further steps for the configuration and the usage require network in most case.
Boot once more under QEMU and use the [Device Manager](https://support.microsoft.com/en-in/help/4028443/windows-10-update-drivers), to ensure all the device drivers, and especially the network card, are installed correctly. Also, as Cloud Hypervisor can introduce new devices, it is advisable to repeat the procedure while booted under Cloud Hypervisor, when the RDP access to the image is functional.
### Windows Special Administration Console (SAC) enablement
SAC provides a text based console access to the Windows guest. As Cloud Hypervisor doesn't implement a VGA adaptor, SAC is an important instrument for the Windows guest management.
Boot the Windows image under QEMU and execute the below commands to permanently enable SAC
```cmd
bcdedit /emssettings emsport:1 emsbaudrate:115200
bcdedit /ems on
bcdedit /bootems on
```
Once SAC is enabled, the image can be booted under Cloud Hypervisor. The SAC prompt will show up
<pre>
Computer is booting, SAC started and initialized.
Use the "ch -?" command for information about using channels.
Use the "?" command for general help.
SAC>
</pre>
To open a console on the guest, the command sequence below can be used
<pre>
SAC>cmd
The Command Prompt session was successfully launched.
SAC>
EVENT: A new channel has been created. Use "ch -?" for channel help.
Channel: Cmd0001
SAC>ch -si 1
</pre>
See also the [links](#Links) section for a more extended SAC documentation.
## Network
This section illustrates the Windows specific corner points for the VM network configuration. For the extended networking guide, including bridging for multiple VMs, follow [networking.md](networking.md).
### Basic Networking
As the simplest option, using `--net tap=` in the Cloud Hypervisor command line will create a `vmtapX` device on the host with the default IPv4 adress `192.168.249.1`. After SAC becomes available, the guest configuration can be set with
This allows for SSH login from a remote machine, for example through the `administrator` user: `ssh administrator@192.168.249.2`. For a more detailed OpenSSH guide, please follow the MSDN article from the [links](#links) section.
## Debugging
The Windows guest debugging process relies heavily on QEMU and [socat](http://www.dest-unreach.org/socat/). The procedure requires two Windows VMs:
- A debugger VM running under QEMU.
- A debuggee, a Windows VM that has been created in the previous steps, running under Cloud Hypervisor or QEMU.
The connection between both guests happens over TCP, whereby on the guest side it is automatically translated to a COM port. Because the VMs are connected through TCP, the debugging infrastructure can be distributed over the network. The serial port, while slowly transferring data, is common enough to support a wide range of cases and tools.
In this excercise, [WinDbg](https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/) is used. Any other debugger of choice with the ability to use serial connection can be used instead.
### Debugger and Debuggee
#### WinDbg VM
For simplicity, the debugger VM is supposed to be only running under QEMU. It will require VGA and doesn't neccessarily depend on UEFI. As an OS, it can carry any supported Windows OS where the debugger of choice can be installed. The simplest way is to follow the image preparation instructions from the previous chapter, but avoid using the OVMF firmware. It is also not required to use VirtIO drivers, whereby it might be useful in some case. Though, while creating the image file for the debugger VM, be sure to choose a sufficient disk size that counts in the need to save the corresponding debug symbols and sources.
To create the debugger Windows VM, the following command can be used:
A non server Windows OS like Windows 10 can be used to carry the debugging tools in the debugger VM.
#### Debuggee VM
The debuggee VM is the one that we've learned to configure and run in the first section. There might be various reasons to debug. For example, there could be an issue in the Windows guest with an emulated device or an included driver. Or, we might want to develop a custom feature like a kernel driver to be available in the guest.
Note, that there are several ways to debug Windows, not all of them need to be enabled at the same time. For example, if developing a kernel module, the only useful options would be to configure for the serial debugging and enable the kernel debug. In that case, any crash or misbehavior in the boot loader or kernel would be ignored. The commands below must be run as administrator on the debuggee guest VM.
##### Turn On Serial Debugging
This will configure the debugging to be enabled and instruct to use the serial port for it.
```cmd
bcdedit /dbgsettings serial debugport:1 baudrate:115200
```
##### Turn On Kernel Debuging
```cmd
bcdedit /debug on
```
##### Turn On Boot Loader Debug
```cmd
bcdedit /bootdebug on
```
##### Turn on boot manager debug
```cmd
bcdedit /set {bootmgr} bootdebug on
```
##### Disable Recovery Screen On Boot Failure
There could be a situation, where a crash is debugged. In such cases, the guest could be left in an inconsistent state. The default Windows behavior would be to boot into the recovery screen, however in some cases it might be not desired. To make Windows ignore failures and always proceed to booting the OS, use the command below:
Note, this VM has the networking enabled. It is needed, because symbols and sources might need to be fetched from a network location.
Also, notice the `-serial` parameter - that's what does the magic on exposing the serial port to the guest while connecting the debugger VM with a client VM through the network. SAC/EMS needs to be disabled in the debugger VM, as otherwise the COM device might be blocked.
Hereafter, WinDbg can be started using a command below:
```cmd
set_NT_DEBUG_PORT=com1
set_NT_DEBUG_BAUD_RATE=115200
windbg -v -d -k
```
Once started, WinDbg will wait for an incoming connection which is going to be initialized by the debuggee VM started in the next section.
#### Invoke the Debuggee VM
##### Under QEMU
Essentially it would be the command like depicted in the guest preparation sections, with a few modifications:
It is to see, that `-serial` parameter is used here, to establish the connection with the debugger VM.
To disable HPET, attach `--no-hpet`. To enable hypervisor reference timer, use `-cpu host,hv-time`. These and other options can be used to achieve better [Hyper-V compatibility](https://archive.fosdem.org/2019/schedule/event/vai_enlightening_kvm/attachments/slides/2860/export/events/attachments/vai_enlightening_kvm/slides/2860/vkuznets_fosdem2019_enlightening_kvm.pdf).
##### Cloud Hypervisor
The `socat` tool is used to establish the QEMU compatible behavior. Here as well, the Cloud Hypervisor command used to run the Windows guest is to be used. Put the command into a shell script:
The reason to pack the command into the shell script is that the command might contain a comma. When using SYSTEM, the shell command can't contain `,` or `!!`.
## Links
- [Fedora VirtIO guide for Windows](https://docs.fedoraproject.org/en-US/quick-docs/creating-windows-virtual-machines-using-virtio-drivers/)
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
