The PL031 RTC provides two features: a real-time counter and an alarm
interrupt. To use the alarm, the driver normally writes a time value
into the match register RTCMR, and when the counter reaches that value
the device triggers the interrupt.
At the moment the implementation ignores programming of the alarm, as
the feature seems rarely used in VMs. However the interrupt is still
triggered arbitrarily when the guest writes to registers, and the line
is never cleared. This really confuses the Linux driver, which loops in
the interrupt handler until Linux realizes that no one is dealing with
the interrupt (200000 unanswered calls) and disables the handler.
One way to fix this would be implementing the alarm function properly,
which isn't too difficult but requires adding some async timer logic
which probably won't ever get used. In addition the device's interrupt
is level-triggered and we don't support level interrupts at the moment,
though we could probably get away with changing this interrupt to edge.
The simplest fix, though, is to just disable the interrupt logic
entirely, so that the alarm function still doesn't work but the guest
doesn't see spurious interrupts.
Add a default() implementation to satisfy clippy's new_without_default
check, since Rtc::new() doesn't take a parameter after this change.
Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
This makes it possible to run cargo test just for the virtio-devices
crate (as long as either KVM or MSHV is specified).
Signed-off-by: Alyssa Ross <hi@alyssa.is>
`std::mem::offset_of` could be used for calculating nested fields, use
this feature to shorten aarch64 reg offset calculation.
Signed-off-by: Ruoqing He <heruoqing@iscas.ac.cn>
When a proper Rust backtrace is printed, the Rust std wants to use the
SYS_getcwd(79) system call to prettify some paths while printing. In
Cloud Hypervisor, this is at least relevant for printing panics or if
a `anyhow::Error` value is printed using `{e:?}` (but not `{e:#?}`).
The syscall cause can be found in `impl fmt::Display for Backtrace {}`
in `library/std/src/backtrace.rs`.
Without this addition, the seccomp violation of the SYS_getcwd (79)
hinders the proper error message including a full backtrace from showing
up. This annoying behaviour already delayed many debugging efforts. With
this fix, things just work. The new syscall itself should be pretty
harmless for normal operation.
```
thread 'vmm' panicked at virtio-devices/src/rng.rs:224:9:
Yikes, things went horribly wrong!
==== Possible seccomp violation ====
Try running with `strace -ff` to identify the cause and open an issue: https://github.com/cloud-hypervisor/cloud-hypervisor/issues/new
[1] 287683 invalid system call (core dumped) RUST_BACKTRACE=full cargo run --bin cloud-hypervisor -- --api-socket --kerne
```
```
thread 'vmm' panicked at virtio-devices/src/rng.rs:224:9:
Yikes, things went horribly wrong!
stack backtrace:
0: 0x557d91286b62 - std::backtrace_rs::backtrace::libunwind::trace::hc20b48b31ee52608
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/../../backtrace/src/backtrace/libunwind.rs:117:9
1: 0x557d91286b62 - std::backtrace_rs::backtrace::trace_unsynchronized::h5d207cd20f193d88
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/../../backtrace/src/backtrace/mod.rs:66:14
...
67: 0x0 - <unknown>
Error: Cloud Hypervisor exited with the following error:
Failed to join on VMM thread: Any { .. }
Debug Info: ThreadJoin(Any { .. })
```
- add any panic, for example into the create or drop function of a
device
- add --seccomp=true|log to analyze the situation
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
On-behalf-of: SAP philipp.schuster@sap.com
Remote server errors are transferred as raw HTTP body. This way,
we lose the nested structured error information.
This is an attempt to retrieve the errors from the HTTP response
and to align the output with the normal error output.
For example, this produces the following chain of errors. Note
that everything after level 0 was retrieved from the HTTP server
response:
```
Error: ch-remote exited with the following chain of errors:
0: http client error
1: Server responded with InternalServerError
2: Error from API
3: The disk could not be added to the VM
4: Failed to validate config
5: Identifier disk1 is not unique
Debug Info: HttpApiClient(ServerResponse(InternalServerError, Some("Error from API<br>The disk could not be added to the VM<br>Failed to validate config<br>Identifier disk1 is not unique")))
```
In case the JSON can't be parsed properly, ch-remote will print:
```
Error: ch-remote exited with the following chain of errors:
0: http client error
X: Can't get remote's error messages from JSON response: EOF while parsing a value at line 1 column 0: body=''
Debug Info: HttpApiClient(ServerResponse(InternalServerError, Some("")))
```
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
On-behalf-of: SAP philipp.schuster@sap.com
With the foundations of each error type implementing std::error::Error,
we can now nicely walk the `.source()` chain and print an error trace.
This commit introduces improved user-facing error printing when:
- Cloud Hypervisor fails with an error
- ch-remote fails (client error)
- ch-remote fails (remote error)
The additional context is a clear improvement in UX for both users and
developers. In the following example, the new behaviour is shown for
a direct invocation of Cloud Hypervisor leading to a failure. This
looks similar for ch-remote.
```
Old Style
`target/release/cloud-hypervisor --api-socket /tmp/chv2.sock --kernel /etc/bootitems/linux/kernel_minimal/stable.bzImage --cmdline console=ttyS0 --serial tty --console off --disk path=img.raw --initramfs /etc/bootitems/linux/initrd_minimal/default`
Error booting VM: VmBoot(LockingError(BlockError(LockDiskImage(AlreadyLocked)))
```
```
`target/release/cloud-hypervisor --api-socket /tmp/chv2.sock --kernel /etc/bootitems/linux/kernel_minimal/stable.bzImage --cmdline console=ttyS0 --serial tty --console off --disk path=img.raw --initramfs
/etc/bootitems/linux/initrd_minimal/default`
Error: Cloud Hypervisor exited with the following chain of errors:
0: Error booting VM
1: The VM could not boot
2: Error locking disk images: Another instance likely holds a lock
3: Cannot lock images of all block devices
4: Failed to get Write lock for disk image: ./img.raw
5: The file is already locked
Debug Info: VmBoot(VmBoot(LockingError(DiskLockError(LockDiskImage { error: AlreadyLocked, lock_type: Write, path: "./raw_disk.bin" })))
```
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
On-behalf-of: SAP philipp.schuster@sap.com
The changes were mostly automatically applied using the following
Python script:
```python
import os, re
for root, _, files in os.walk("."):
for f in files:
if not f.endswith(".rs"):
continue
p = os.path.join(root, f)
with open(p, "r", encoding="utf-8") as file:
lines = file.readlines()
changed = False
for i in range(len(lines) - 1):
if re.search(r'#\[error\(".*: \{0[^}]*\}"\)\]', lines[i]) and "#[source]" in lines[i + 1].strip():
lines[i] = re.sub(r': \{0[^}]*\}"\)\]', '")]', lines[i])
changed = True
if changed:
with open(p, "w", encoding="utf-8") as file:
file.writelines(lines)
print("Fixed:", p)
```
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
On-behalf-of: SAP philipp.schuster@sap.com
# Conflicts:
# vmm/src/api/http/mod.rs
This was caught by the nightly compiler during cargo fuzz build.
error: lifetime flowing from input to output with different syntax can be confusing
--> /home/runner/work/cloud-hypervisor/cloud-hypervisor/hypervisor/src/arch/x86/emulator/mod.rs:493:26
|
493 | pub fn new(platform: &mut dyn PlatformEmulator<CpuState = T>) -> Emulator<T> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ----------- the lifetime gets resolved as `'_`
| |
| this lifetime flows to the output
|
= note: `-D mismatched-lifetime-syntaxes` implied by `-D warnings`
= help: to override `-D warnings` add `#[allow(mismatched_lifetime_syntaxes)]`
help: one option is to remove the lifetime for references and use the anonymous lifetime for paths
|
493 | pub fn new(platform: &mut dyn PlatformEmulator<CpuState = T>) -> Emulator<'_, T> {
Signed-off-by: Jinank Jain <jinankjain@microsoft.com>
Since it is used by multiple components at this point, it is better to
move it to workspace level dependency.
Signed-off-by: Jinank Jain <jinankjain@microsoft.com>
This commit adds the warning to deprecate the SGX support with the
intention to remove the support from code base in two release cycles.
See: #6960
Signed-off-by: Bo Chen <bchen@crusoe.ai>
This streamlines the Error implementation in the Cloud Hypervisor code
base to match the remaining parts so that everything follows the agreed
conventions. These are leftovers missed in the previous commits.
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
On-behalf-of: SAP philipp.schuster@sap.com
No sane guest OS will allow hotplugging all cpus.
However, the REST API currently allows specifying
`ch-remote resize --cpus 0`.
On Linux, we can then observe the following error in the kernel log:
processor cpu0: Offline failed.
Subsequent resize commands via ch-remote will then fail with
VcpuPendingRemovedVcpu because the removal of cpu0 was never
successful.
Fix this by disallowing resizing to zero vCPUs.
Signed-off-by: Thomas Prescher <thomas.prescher@cyberus-technology.de>
On-behalf-of: SAP thomas.prescher@sap.com
For use in QEMU, I would like GuestMemoryRegion to return a BitmapSlice
instead of a &Bitmap. This adds some flexibility that QEMU needs in
order to support a single global dirty bitmap that is sliced by the
various GuestMemoryRegions.
However, this removes access to the methods of AtomicBitmap, and in
particular reset() and get_and_reset(). Fortunately, cloud-hypervisor
always uses GuestMemoryMmap, and therefore `region` is known to be a
&GuestRegionMmap. Dereferencing it returns the MmapRegion to which the
bitmap is attached, thus calling MmapRegion::bitmap(); this has the
same effect as `<GuestRegionMmap as GuestRegion>::bitmap()`, and works
both with or without https://github.com/rust-vmm/vm-memory/pull/324.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
This streamlines the code base to follow best practices for
error handling in Rust: Each error struct implements
std::error::Error (most due via thiserror::Error derive macro)
and sets its source accordingly.
This allows future work that nicely prints the error chains,
for example.
So far, the convention is that each error prints its
sub error as part of its Display::fmt() impl.
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
On-behalf-of: SAP philipp.schuster@sap.com
This streamlines the code base to follow best practices for
error handling in Rust: Each error struct implements
std::error::Error (most due via thiserror::Error derive macro)
and sets its source accordingly.
This allows future work that nicely prints the error chains,
for example.
So far, the convention is that each error prints its
sub error as part of its Display::fmt() impl.
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
On-behalf-of: SAP philipp.schuster@sap.com
Allow tap interfaces to be configured with an IPv6 address. The change
is fairly straightforward: we need to update the API types and CLI
parsing to accept either an IPv6 or IPv4 and then match on the IP
address type when the tap device is configured.
For IPv6 addresses, the netmask (prefix) must be provided at the same
time as the address itself (in the SIOCSIFADDR ioctl). They cannot be
configured separately. So we remove the separate "set_netmask" function
and convert "set_ip_addr" to also accept a netmask. For IPv4 addresses,
the IP address and netmask were already always set together, so this
should have no functional impact for users of IPv4 addresses.
Signed-off-by: Gregory Anders <ganders@cloudflare.com>
The DeviceManagerError type is among the types missing the Error trait
so far. To streamline the code and to simplify usage on higher levels
of this error type, this type now implements Display and Error.
As not all variant values are Error yet, `#[source]` is not everywhere
where it could be. This is done in the next commits.
The high level goal is: Enable future work to improve the error output
of cloud hypervisor.
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
On-behalf-of: SAP philipp.schuster@sap.com
# What
This commit introduces file-based advisory locking for the files backing
up the block devices by using the fcntl() syscall with OFD locks. The
per-open-file-descriptor (OFD) locks are more robust than traditional
POSIX locks (F_SETLK) as they are not tied to process IDs and avoid
common issues in multithreaded or multi-fd scenarios [1]. Therefore,
we don't use `std::fs::File::try_lock()`, which is backed by F_SETLKW.
The locking mechanism is aware of the `readonly` property and allows
`n` readers or `1` writer (exclusive mode).
As the locks are advisory, multiple cloud-hypervisor processes can
prevent themselves from writing to the same file. However, this is not
a system-wide file-system level locking mechanism preventing to open()
a file.
The introduced new locking mechanism does not cover vhost-user devices.
# Why
To prevent misconfiguration and improve safety, it is good practice to
protect disk image files with a locking mechanism. Experience and common
best practices suggest that advisory locks are preferable over mandatory
locks due to better compatibility and fewer pitfalls (in fs space).
The introduced functionality is aligned with the approach taken by
QEMU [0], and is also recommended in [1].
# Implementation Details
We need to ensure that not only normal operation keeps working but also
state save/resume and live-migration. Especially for live migration,
it is crucial that the sender VMM releases the locks when the VM stops
so the receiver VMM can acquire them right after that.
Therefore, the locking and releasing happen directly on the block
device struct. The device manager knows all block devices and can
forward requests to these types.
Last but not least, this commit uses on explicit lock acquiring
but implicit lock releasing (FD close). It only explicitly releases
the locks where this integrates more smoothly into the existing
code.
# Testing
I tested
- normal operation
- state save/resume,
- device hot plugging,
- and live-migration
with read/shared and write/exclusive locks.
One can use the `fcntl-tool` to test if locks are actually acquired
or released [2].
# Links
[0] 825b96dbce/util/osdep.c (L266)
[1] https://apenwarr.ca/log/20101213
[2] https://crates.io/crates/fcntl-tool
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
On-behalf-of: SAP philipp.schuster@sap.com
Fixing the following clippy issue using `cargo clippy --fix`:
error: variables can be used directly in the `format!` string
--> build.rs:25:27
|
25 | version.push_str(&format!("-{}", extra_version));
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#uninlined_format_args
Signed-off-by: Bo Chen <bchen@crusoe.ai>
It seems like address allocation has been spread into different files
and different location for x86 vs ARM. This makes it hard to follow the
code. Thus, unify it a single location which satisfies all the
requirement.
Signed-off-by: Jinank Jain <jinankjain@microsoft.com>
It just simplifies code and improves the code read-ability without much
affecting the boot performance.
Signed-off-by: Jinank Jain <jinankjain@microsoft.com>
This will satisfy the requirement of MSHV i.e., setting the GICD base
address before initializing the VM.
Signed-off-by: Jinank Jain <jinankjain@microsoft.com>
For MSHV guests, we would need interrupt controller to be initialized
before the VM gets initialized. This is because we are registering the
base address of GIC distributor with MSHV as part of interrupt
controller initialization workflow. And MSHV mandates that this property
is set before we initialize the VM.
Signed-off-by: Jinank Jain <jinankjain@microsoft.com>
Since this would be used in other places inside the hypervisor and
hypervisor crate cannot take a dependency on arch crate, as that creates
cyclic dependency.
Signed-off-by: Jinank Jain <jinankjain@microsoft.com>
Along with also bump the vfio-bindings crates to use the latest
mshv-bindings.
There is a breaking change in the new mshv crate which requires an
additional step to initialize vm after creating it.
Signed-off-by: Jinank Jain <jinankjain@microsoft.com>
