In this way, we can cover the scenario where a VM with hotplugged net
device using FDs can work properly with reboot.
Signed-off-by: Bo Chen <chen.bo@intel.com>
The custom 'clone' duplicates 'preserved_fds' so that the validation
logic can be safely carried out on the clone of the VmConfig.
The custom 'drop' ensures 'preserved_fds' are safely closed when the
holding VmConfig instance is destroyed.
Signed-off-by: Bo Chen <chen.bo@intel.com>
Preserved FDs are the ones that share the same life-time as its holding
VmConfig instance, such as FDs for creating TAP devices.
Preserved FDs will stay open as long as the holding VmConfig instance is
valid, and will be closed when the holding VmConfig instance is destroyed.
Signed-off-by: Bo Chen <chen.bo@intel.com>
When neither serial nor console are connected to the tty,
cloud-hypervisor shouldn't touch the tty at all. One way in which
this is annoying is that if I am running cloud-hypervisor without it
using my terminal, I expect to be able to suspend it with ^Z like any
other process, but that doesn't work if it's put the terminal into raw
mode.
Instead of putting the tty into raw mode when a VM is created or
restored, do it when a serial or console device is created. Since we
now know it can't be put into raw mode until the Vm object is created,
we can move setting it back to canon mode into the drop handler for
that object, which should always be run in normal operation. We still
also put the tty into canon mode in the SIGTERM / SIGINT handler, but
check whether the tty was actually used, rather than whether stdin is
a tty. This requires passing on_tty around as an atomic boolean.
I explored more of an abstraction over the tty — having an object that
encapsulated stdout and put the tty into raw mode when initialized and
into canon mode when dropped — but it wasn't practical, mostly due to
the special requirements of the signal handler. I also investigated
whether the SIGWINCH listener process could be used here, which I
think would have worked but I'm hesitant to involve it in serial
handling as well as conosle handling.
There's no longer a check for whether the file descriptor is a tty
before setting it into canon mode — it's redundant, because if it's
not a tty it just won't respond to the ioctl.
Tested by shutting down through the API, SIGTERM, and an error
injected after setting raw mode.
Signed-off-by: Alyssa Ross <hi@alyssa.is>
If the VM is shut down, either it's going to be started again, in
which case we still want to be in raw mode, or the process is about to
exit, in which case canon mode will be set at the end of main.
Signed-off-by: Alyssa Ross <hi@alyssa.is>
Having PMU in guests isn't critical, and not all hardware supports
it (e.g. Apple Silicon).
CpuManager::init_pmu already has a fallback for if PMU is not
supported by the VCPU, but we weren't getting that far, because we
would always try to initialise the VCPU with KVM_ARM_VCPU_PMU_V3, and
then bail when it returned with EINVAL.
Signed-off-by: Alyssa Ross <hi@alyssa.is>
Cloud Hypervisor's vhost-user implementation will reconnect if it gets
disconnected from the backend. That means connections happen inside
the vhost-user seccomp sandbox, so all syscalls used in reconnecting
have to be allowed in that sandbox.
clock_nanosleep is used by Glibc, and nanosleep is used by musl.
Signed-off-by: Alyssa Ross <hi@alyssa.is>
The PR #2333 added I/O rate limiter on block device, with some options
in `DiskConfig`. And the PR #2401 added rate limiter on virtio-net
device with same options, but it still throws `Error::ParseDisk`.
This commit fixes it with correct values.
Fixes: #2401
Signed-off-by: Yu Li <liyu.yukiteru@bytedance.com>
Unused import 'xml' is redefined at:
> xml = ET.ElementTree(element=svg)
Hence, remove unused xml import.
Signed-off-by: Ruslan Mstoi <ruslan.mstoi@intel.com>
Any error from the backend or from the protocol handling code will now
result in an error reply being sent. This is cleanly achieved by
splitting the command handling out into its own method and using the
Rust Result<> based error handling to trigger the generation of the
error reply.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This allows the implementation of PCI devices in a different process
using the vfio-user protocol.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Replace the use of an enum with a bitfield representation which means
that is now possible to logical OR flags together.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Once error occur, vcpu thread may exit, this should
be critical event for the whole VM, we should fire
exit event and set vcpu state.
If we don't set vcpu state, the shutdown process
will hang at signal_thread, which is waiting the
vcpu state to change.
Signed-off-by: Yong He <alexyonghe@tencent.com>
This test (which relies on nesting) is failing on the VFIO worker. The tests that use the
dedicated hardware pass fine.
See: #5190
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
We need to provide valid FDs while creating 'NetConfig' instances even
for unit tests. Closing invalid FDs would cause random unit test
failures.
Also, two identical 'NetConfig' instances are not allowed any more,
because it would lead to close the same FD twice. This is consistent
with the fact that a clone of a "NetConfig" instance is no
longer *equal* to the instance itself.
Fixes: #5203
Signed-off-by: Bo Chen <chen.bo@intel.com>
These are owned by the config (and are duplicated before being used to
create the `Tap` for the virtio-net device.)
By implementing Drop on NetConfig we have issues with moving out of
members that don't implement the Copy trait. This requires a small
adjustment to the unit tests that use the Default::default() function.
Fixes: #5197
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The custom version duplicates any FDs that have been provided so that
the validation logic used on hotplug, which takes a clone of the config,
can be safely carried out.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This code is indentical to what is in this repository. When a release
gets made we can then switch to that.
Fixes: #5122
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
If swtpm becomes unresponsive, guest gets blocked at "recvmsg" on tpm's
data FD. This change adds a timeout to the data fd socket. If swtpm
becomes unresponsive guest waits for "timeout" (secs) and continues to
run after returning an I/O error to tpm commands.
Signed-off-by: Praveen K Paladugu <prapal@linux.microsoft.com>
The updated image is configured in a same way as the previously used
2019, it has same
- Credentials
- Services configured, like SAC, SSH, RDP
- Size
All the Windows updates are applied so the state is current to the date.
Also, the latest stable version 0.1.229 of the VirtIO Windows drivers
is installed.
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
We can ideally defer the address space allocation till we start the
vCPUs for the very first time. Because the VM will not access the memory
until the CPUs start running. Thus there is no need to allocate the
address space eagerly and wait till the time we are going to start the
vCPUs for the first time.
Signed-off-by: Jinank Jain <jinankjain@microsoft.com>
This hypervisor leaf includes details of the TSC frequency if that is
available from KVM. This can be used to efficiently calculate time
passed when there is an invariant TSC.
TEST=Run `cpuid` in the guest and observe the frequency populated.
Fixes: #5178
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Updates include:
- Add references to 'TDX Tools'
- Expand instructions on buidling and using TDShim
- Add version information of guest/host kernel, TDVF, TDShim being tested
Signed-off-by: Bo Chen <chen.bo@intel.com>
This is required for booting Linux:
From: https://lore.kernel.org/all/20221028141220.29217-3-kirill.shutemov@linux.intel.com/
"""
Virtualization Exceptions (#VE) are delivered to TDX guests due to
specific guest actions such as using specific instructions or accessing
a specific MSR.
Notable reason for #VE is access to specific guest physical addresses.
It requires special security considerations as it is not fully in
control of the guest kernel. VMM can remove a page from EPT page table
and trigger #VE on access.
The primary use-case for #VE on a memory access is MMIO: VMM removes
page from EPT to trigger exception in the guest which allows guest to
emulate MMIO with hypercalls.
MMIO only happens on shared memory. All conventional kernel memory is
private. This includes everything from kernel stacks to kernel text.
Handling exceptions on arbitrary accesses to kernel memory is
essentially impossible as handling #VE may require access to memory
that also triggers the exception.
TDX module provides mechanism to disable #VE delivery on access to
private memory. If SEPT_VE_DISABLE TD attribute is set, private EPT
violation will not be reflected to the guest as #VE, but will trigger
exit to VMM.
Make sure the attribute is set by VMM. Panic otherwise.
There's small window during the boot before the check where kernel has
early #VE handler. But the handler is only for port I/O and panic as
soon as it sees any other #VE reason.
SEPT_VE_DISABLE makes SEPT violation unrecoverable and terminating the
TD is the only option.
Kernel has no legitimate use-cases for #VE on private memory. It is
either a guest kernel bug (like access of unaccepted memory) or
malicious/buggy VMM that removes guest page that is still in use.
In both cases terminating TD is the right thing to do.
"""
With this change Cloud Hypervisor can boot the current Linux guest
kernel.
Reported-By: Jiaqi Gao <jiaqi.gao@intel.com
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Do the following:
1. Use from_be_bytes to drop mutable slices.
2. Check for the exact buffer size throughout.
3. Simplify ptm_to_request where possible.
4. Make error messages style consistent.
Fix a typo in code comment while at it.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
There is no guarantee that the write can send the whole buffer at once.
In those rare occasions, we should return a sensible error.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
The largest possible PTM response is only 16 bytes. Size the output
buffer correctly.
In the socket read function, rely on the caller to provide a
sufficiently large buffer. That eliminates another large stack variable.
In total this saves almost 8KB stack space.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Change "thead" to "thread".
Also make sure the two messages are distinguishable by adding "vmm" and
"vm" prefix.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
The number of aligned operations can not be larger than the number of
descriptors. Initializing the capacity to 1 is good enough per the
observation that most of time there is only one data descriptor in a
given request.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Before Linux v6.0, AArch64 didn't support "socket" in "cpu-map"
(CPU topology) of FDT.
We found that clusters can be used in the same way of sockets. That is
the way we implemented the socket settings in Cloud Hypervisor. But in
fact it was a bug.
Linux commit 26a2b7 fixed the mistake. So the cluster nodes can no
longer act as sockets. And in a following commit dea8c0, sockets were
supported.
This patch fixed the way to configure sockets. In each socket, a default
cluster was added to contain all the cores, because cluster layer is
mandatory in CPU topology on AArch64.
This fix will break the socket settings on the guests where the kernel
version is lower than v6.0. In that case, if socket number is set to
more than 1, the kernel will treat that as FDT mistake and all the CPUs
will be put in single cluster of single socket.
The patch only impacts the case of using FDT, not ACPI.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Fix lowercase label to avoid "mkfs.fat: Warning: lowercase labels
might not work properly on some systems".
Signed-off-by: Ravi kumar Veeramally <ravikumar.veeramally@intel.com>
This patch adds a global execution timeout to the Jenkinsfile to avoid
infinite pending Jenkins pipelines, such as when certain worker nodes
are not available. The global execution timeout is now set to 4 hours
which is derived from total timeout of our longest stage (e.g. the
`Worker build`).
Fixes: #5148
Signed-off-by: Bo Chen <chen.bo@intel.com>
As a first time user of cloud-hypervisor and Rust environment
you get build errors with out this.
Signed-off-by: Ravi kumar Veeramally <ravikumar.veeramally@intel.com>
Right now integration test fails during the test run if
/dev/mshv or /dev/kvm does not exist. We should not
progress and exit early if not present.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Make the code more idiomatic by wrapping the actual size configured in
the returning Result type. This further allows simplifying
get_buffer_size.
The debug message in startup_tpm is more useful if it prints out the
actual size than the wanted size.
No functional change.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
In order to comply with latest TDX version, we rely onto the branch
kvm-upstream-2022.08.07-v5.19-rc8 from https://github.com/intel/tdx
repository. Updates are based on changes that happened in
arch/x86/include/uapi/asm/kvm.h headers file.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Dependabot will create a branch on the repo for it's updates this
triggers the release action (because it's the same event as a tag) which
will then fail leading to dependabot PRs not being automerged. Instead
only run the release check test on PRs or tag creation.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
KSM doesn't work with MSHV stack since guest memory is pinned
(`pin_user_pages`) and pinned pages cannot be merged.
So, don't run the test for mshv.
Signed-off-by: Anirudh Rayabharam <anrayabh@linux.microsoft.com>
This makes it much more convenient to build binaries for profiling
without having to modify the Cargo.toml file.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
A few breaking changes:
1. `-vvv` needs to be written as `-v -v -v`.
2. `--disk D1 D2` and others need to be written as `--disk D1 --disk D2`.
3. `--option=value` needs to be written as `--option value`
Change integration tests to adapt to the breaking changes.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Since argh does not support `--option=value`, we need to change the
integration test code to become `--option value`.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Since we run "cargo clean" before running the aarch64 build we need to
create the release and upload the x86-64 assets before the clean.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Some crates don't need it at all.
Some crates are using it for a simple functionality which can be
replaced easily.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
This was causing issues with the release build process but we now have a
fix to clean residual build assets.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Adjust the release workflow to move the conditional check on the tag
creation into the steps that create the release/upload the assets.
This allows us to ensure we're always in a releaseable state.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This reverts commit 998fb48ff2.
This is breaking the release build process (cross build for aarch64
musl) so temporarily reverting until we can identify the cause to allow
the release proceeed.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This change is important to do a proper resource cleanup. We decided
to do this repetitive approach as VirtioCommon can't implement Drop
without major changes to the corresponding code. Also, devices such as
Net can't easily use the epoll_threads-abstraction from VirtioCommon as
it has multiple threads with different semantics.
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
Add new configuration for offloading features, including
Checksum/TSO/UFO, and set these offloading features as
enabled by default.
Fixes: #4792.
Signed-off-by: Yong He <alexyonghe@tencent.com>
There was an unnecessary change in previous PR #5077.
This is the follow-up clean up patch.
Right now there is no use case of the drive of
Eq and PartialEq.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
Add new latency counters for virtio-block device, including
minimal latency, maximal latency, and average latency for block
read and write.
The average latency is calculated based on cumulative average.
Signed-off-by: Yong He <alexyonghe@tencent.com>
MSHV does not require to ensure MMIO/PIO exits complete
before pausing. This patch makes sure the above requirement
by checking the hypervisor type run-time.
Fixes#5037
Signed-off-by: Muminul Islam <muislam@microsoft.com>
e.g. on QEMU on KVM:
cloud-hypervisor: 17.079406ms: <vmm> INFO:arch/src/x86_64/mod.rs:565 -- Running under nested virtualisation. Hypervisor string: KVMKVMKVM
Or under Azure:
cloud-hypervisor: 3.881263ms: <vmm> INFO:arch/src/x86_64/mod.rs:565 -- Running under nested virtualisation. Hypervisor string: Microsoft Hv
Fixes: #5067
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
For the synchronous backends efficiently preserve the order for
completion requests through the use of VecDequeue. Preserving the order
is not required but is beneficial as it matches the existing
optimisation that looks to match completions and requests.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Rather than aggregate the completion list into an intermediate vector
instead adjust the API to provide one completion item at a time.
With DHAT this shows the number of heap allocations has decreased.
Before:
dhat: Total: 623,852 bytes in 8,157 blocks
After:
dhat: Total: 380,444 bytes in 3,469 blocks
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The number of buffers in the request is usually one so by using SmallVec
the number of heap allocations can be significantly reduced.
DHAT reports:
Before:
dhat: Total: 1,166,412 bytes in 40,383 blocks
After:
dhat: Total: 623,852 bytes in 8,157 blocks
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Rather than passing the vector of iovecs for the I/O to act on pass a
reference to the slice of values inside them. This removes the explicit
container type from the API allowing the use of e.g. SmallVec.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This functionality has been obsoleted by our native support for
hugepages and shared memory.
See: #5082
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The number of entries in the refcount table was incorrectly calculated
given there was no need for dividing the number of refblock clusters.
The number of refblock clusters is the number of entries in the refcount
table.
Suggested-by: lv_mz <lv.mengzhao@zte.com.cn>
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
During analysis of the asynchrous block I/O handling it was observed
that the majority of the time the completion events occur in the same
order as submissions. Further the maximum number of inflight requests
during the boot time is much lower than the size of the queue.
Through the use of a double ended queue (VecDequeue) with a reasonable
pre-allocation capacity we can have O(1) allocation free addition of
items to the list of inflight requests and mostly O(1) matching of
completed requests to submissions.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Since SGX testing doesn't rely on a custom guest image anymore, there's
no need to keep the custom filename around as it's already not in use.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
When using DHAT[1] for analysis the amount heap allocations change from:
dhat: Total: 3,186,536 bytes in 41,452 blocks
to
dhat: Total: 1,059,816 bytes in 34,747 blocks
When running against virtio-block; this still more allocations than
virtio-pmem but a significant improvement without any cost.
[1] https://docs.rs/dhat/latest/dhat/
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
With this change, all log messages will have the same width for the
timestamp. The number of ms is rounded to 6 decimal places.
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
To align the logging messages with the rest of the code, this
message should be aligned with another similar occurrence in
epoll_helper.rs
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
Some crates don't need it at all.
Some crates are using it for a simple functionality which can be
replaced easily.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
While measuring UDP PPS, we saturate the link, so there are packets
lost. We only account for the packets that are not lost.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Change fio_ops to fio_control and add a new field to indicate whether it
should report bandwidth or IOPS.
All existing tests are bandwidth tests. Adapt the code accordingly. Add
a set of new tests to report IOPS.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
This OS is EOL this year and is well tested by the Rust Hypervisor
Firmware CI so there is no need to duplicate this effort.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
With this bump there was a change in one of the externally exposed
variable. Thus, the use of that variable in CLH must be adjusted
accordingly.
Signed-off-by: Jinank Jain <jinankjain@microsoft.com>
The double underscore made it different from how other projects would
name this particular macro.
No functional change.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
There is duplicated code when handlin queue events in handle_event()
refactor and introduce a new helper function.
Signed-off-by: Hao Xu <howeyxu@tencent.com>
Between musl and glibc there is a difference in the signature of the
ioctl libc function. Use an anonymous cast to force the type coversion.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
after testing the build briefly, it seems upstream tianocore EDK2 works fine,
no need for the separate cloud-hypervisor edk2 repo.
Signed-off-by: Claudio Fontana <claudio.fontana@gmail.com>
Remove from the documentation and API definition but continue support
using the field (with a deprecation warning.)
See: #4837
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Since the refactoring of the vm-migration crate broke the backward
compatibility, we must disable the live upgrade tests until next
release.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This simplifies the Snapshot creation as we expect a SnapshotData to be
provided most of the time.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The information about the identifier related to a Snapshot is only
relevant from the BTreeMap perspective, which is why we can get rid of
the duplicated identifier in every Snapshot structure.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
There's no reason to carry a HashMap of SnapshotDataSection per
Snapshot. And given we now provide at most one SnapshotDataSection per
Snapshot, there's no need to keep the id part of the SnapshotDataSection
structure.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Without breaking the former way of declaring them. This is simply based
on the presence of the GUID TDX Metadata offset. If not present, we
consider the firmware is quite old and therefore we fallback onto the
previous way to expose memory resources.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In particular update to latest linux-loader release and point to latest
vfio repository for both crates hosted there.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The preferred way of retrieving the offset where to find the TDVF
descriptor structure is by going through a table of GUIDs that can be
found at a specific offset in the firmware file. If the expected GUIDs
can't be found, we can fallback onto the former way, which is to read
directly the value at a specific offset in the file.
This patch implements the new mechanism without breaking compatibility
for older firmwares as it keeps supporting the previous mechanism.
As a reference, here is the documentation from the EDK2 code, and
particularly from the OvmfPkg/ResetVector/Ia16/ResetVectorVtf0.asm file:
```
GUIDed structure. To traverse this you should first verify the
presence of the table footer guid
(96b582de-1fb2-45f7-baea-a366c55a082d) at 0xffffffd0. If that
is found, the two bytes at 0xffffffce are the entire table length.
The table is composed of structures with the form:
Data (arbitrary bytes identified by guid)
length from start of data to end of guid (2 bytes)
guid (16 bytes)
so work back from the footer using the length to traverse until you
either find the guid you're looking for or run off the beginning of
the table.
```
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The datatype used for the ioctl() C library call is different between it
and the glibc toolchains. The easiest solution is to have the compiler
type cast to type of the parameter.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The coredump functionality is only implemented for x86_64 so it should
only be compiled in there.
Fixes: #4964
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
TDX was broken by the recent refactoring moving the vCPU creation
earlier than before. The simple and correct way to fix this problem is
by moving the TDX initialization right before the vCPUs creation. The
rest of the TDX setup can remain where it is.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This removes the storage of the GuestMemoryMmap on the CpuManager
further allowing the decoupling of the CpuManager from the
MemoryManager.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
When configuring the vCPUs it is only necessary to provide the guest
memory when booting fresh (for populating the guest memory). As such
refactor the vCPU configuration to remove the use of the
GuestMemoryMmap stored on the CpuManager.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Thanks to the new way of restoring Vm, we can now create the Vm object
directly with the appropriate VmState rather than having to patch it at
a later time.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
No need to provide a boolean to know if the VM is being restored given
we already have this information from the Option<Snapshot>.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Now the entire codebase has been moved to the new restore design, we can
complete the work by creating a dedicated restore() function for the Vm
object and get rid of the method restore() from the Snapshottable trait.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
On aarch64 there is no modification of the app struct however
refactoring to remove this would be very intrusive.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The snapshot and restore of AArch64 Gic was done in Vm. Now it is moved
to DeviceManager.
The benefit is that the restore can be done while the Gic is created in
DeviceManager.
While the moving of state data from Vm snapshot to DeviceManager
snapshot breaks the compatability of migration from older versions.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Given the recent factorization that happened in vm.rs, we're now able to
merge Vm::new_from_snapshot with Vm::new.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This moves the devices creation out of the dedicated restore function
which will be eventually removed.
This factorizes the creation of all devices into a single location.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This allows the clock restoration to be moved out of the dedicated
restore function, which will eventually be removed.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Based on all the work that has already been merged, it is now possible
to fully move DeviceManager out of the previous restore model, meaning
there's no need for a dedicated restore() function to be implemented
there.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Every Vcpu is now created with the right state if there's an available
snapshot associated with it. This simplifies the restore logic.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Following the new restore design, it is not appropriate to set every
virtio device threads into a paused state after they've been started.
This is why we remove the line of code pausing the devices only after
they've been restored, and replace it with a small patch in every virtio
device implementation. When a virtio device is created as part of a
restored VM, the associated "paused" boolean is set to true. This
ensures the corresponding thread will be directly parked when being
started, avoiding the thread to be in a different state than the one it
was on the source VM during the snapshot.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
To synthesize the interactions between the virtio-net device and the tap
interface, the fuzzer utilizes a pair of unix domain sockets: one socket
(e.g. the dummy tap frontend) is used to construct the 'net_util::Tap'
instance for creating a virtio-net device; the other socket (e.g. the
dummy tap backend) is used in a epoll loop for handling the tx and rx
requests from the virtio-net device.
Signed-off-by: Bo Chen <chen.bo@intel.com>
Three functions are added:
* 'Tap::new_for_fuzzing()' a custom constructor that creates a dummy
`Tap` interface directly from `File` backed by Unix domain socket;
* 'Tap::mtu()' a custom function that returns hard-coded mtu;
* 'Net::wait_for_epoll_threads()'.
Two functions are reused with modifications to work with the dummy 'Tap'
interface:
* 'Net::new_with_tap()' is made public for fuzzing;
* 'Net::activate()' is modified to not call into 'Tap::set_offload()'
for fuzzing.
Signed-off-by: Bo Chen <chen.bo@intel.com>
To me the most logical place to document the policy is right next to the
version itself.
Fixes: #4318
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Now that VirtioPciDevice, VfioPciDevice and VfioUserPciDevice have all
been moved to the new restore design, there's no need to keep the old
way around, therefore the restore() implementations for MsiConfig,
MsixConfig and PciConfiguration can be removed.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This is some preliminatory work for moving both VfioUser and Vfio to the
new restore design.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This integer overflow was triggered with fuzzing on the virtio-net
device. The integer overflow is from the wrong assumption that the
packets read from or written to the tap device is always larger than the
size of a virtio-net header.
Signed-off-by: Bo Chen <chen.bo@intel.com>
Moving the Ioapic object to the new restore design, meaning the Ioapic
is created directly with the right state, and it shares the same
codepath as when it's created from scratch.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Extend the existing `custom-image.md` document with a new section on how
to create a custom image that contains NVIDIA drivers that are required
for our VFIO baremetal CI.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Re-enable the VFIO integration now the machine is back online.
The image has been updated to rely on Ubuntu 22.04 (Jammy) and it's
smaller given only the NVIDIA drivers along with the nvidia-smi tool are
installed.
The test to verify the GPU is functional has been simplified given it
only relies on nvidia-smi to validate it has been able to find the Tesla
T4 card, meaning the associated driver was loaded correctly.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This uncouples it from the features used for building the binary under
test allowing it to use the default build features.
This change also removes the feature control from the test scripts where
it was never used (e.g. run_integration_tests_sgx.sh)
This allows the combined binary to be used for all testing but allows
the disabling of tests known not to work under mshv.
Fixes: #4915
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This will help with debugging issues like #4917 since we can find which
ioctl causes the error.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Creating a dedicated Result type for VirtioPciDevice, associated with
the new VirtioPciDeviceError enum. This allows for a clearer handling of
the errors generated through VirtioPciDevice::new().
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The code for restoring a VirtioPciDevice has been updated, including the
dependencies VirtioPciCommonConfig, MsixConfig and PciConfiguration.
It's important to note that both PciConfiguration and MsixConfig still
have restore() implementations because Vfio and VfioUser devices still
rely on the old way for restore.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This allows the unification of the same testing methodology with aarch64
and removes a user of the Bionic image.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In the new process, `device::Gic::new()` covers additional actions:
1. Creating `hypervisor::vGic`
2. Initializing interrupt routings
The change makes the vGic device ready in the beginning of
`DeviceManager::create_devices()`. This can unblock the GIC related
devices initialization in the `DeviceManager`.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Moving the creation of the vCPUs before the DeviceManager gets created
will allow for the aarch64 vGIC to be created before the DeviceManager
as well in a follow up patch. The end goal being to adopt the same
creation sequence for both x86_64 and aarch64, and keeping in mind that
the vGIC requires every vCPU to be created.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Split the vCPU creation into two distincts parts. On the one hand we
create the actual Vcpu object with the creation of the hypervisor::Vcpu.
And on the other hand, we configure the existing Vcpu, setting registers
to proper values (such as setting the entry point).
This will allow for further work to move the creation earlier in the
boot, so that the hypervisor::Vcpu will be already created when the
DeviceManager gets created.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
The CpuManager is now created before the DeviceManager. This is required
as preliminary work for creating the vCPUs before the DeviceManager,
which is required to ensure both x86_64 and aarch64 follow the same
sequence.
It's important to note the optimization for faster PIO accesses on the
PCI config space had to be removed given the VmOps was required by the
CpuManager and by the Vcpu by extension. But given the PciConfigIo is
created as part of the DeviceManager, there was no proper way of moving
things around so that we could provide PciConfigIo early enough.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This optimisation provided some peformance improvement when measured by
perf however when considered in terms of boot time peformance this
optimisation doesn't have any impact measurable using our
peformance-metrics tooling.
Removing this optimisation helps simplify the VMM internals as it allows
the reordering of the VM creation process permitting refactoring of the
restore code path.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This panic was triggered with fuzzing on the virtio-net device. This
commits handles the error explicitly to avoid the panic, which also
makes the fuzzer happy (as panic is treated as bugs).
Signed-off-by: Bo Chen <chen.bo@intel.com>
If the KVM version is too old (pre Linux 5.7) then fetch the CPUID
information from the host and use that in the guest. We prefer the KVM
version over the host version as that would use the CPUID for the
running CPU vs the CPU that runs this code which might be different due
to a hybrid topology.
Fixes: #4918
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Fixes: #4678
Currently release build is done on kvm feature only,
that makes live upgrade test on MSHV failing since
it does not find /dev/kvm. As Cloud-Hypervisor
supports both kvm and mshv in a single binary we should
make the release build with both KVM/MSHV feature enabled.
That way live upgrade test does not fail on MSHV.
Signed-off-by: Muminul Islam <muislam@microsoft.com>
The jammy disk image has a new enough kernel to support SGX and if we
rely on just the CPUid information (which is sufficient) then we can use
the regular jammy test image for testing.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This reverts commit 345e65c9c2.
The AArch64 server maintenance is done. Now we can enable the CI again.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
This device emulates a tpm device in cloud-hypervisor. Read and write support
to miminal set of control registers is supported.
Non-control commands will be processed as reads and write to
CRB_DATA_BUFFER. Reads and Writes to this data buffer are forwarded to
swtpm via the configured data_fd.
Signed-off-by: Praveen K Paladugu <prapal@linux.microsoft.com>
Co-authored-by: Sean Yoo <t-seanyoo@microsoft.com>
Emulator module adds methods required to communicate with swtpm
over Ctrl and Data channels.
Signed-off-by: Praveen K Paladugu <prapal@linux.microsoft.com>
Co-authored-by: Sean Yoo <t-seanyoo@microsoft.com>
Add SocketDev struct. Methods in SocketDev will be used to read & write
to Ctrl channel created by swtpm.
Signed-off-by: Praveen K Paladugu <prapal@linux.microsoft.com>
Co-authored-by: Sean Yoo <t-seanyoo@microsoft.com>
Add structures and related methods to process Ctrl requests and responses
from swtpm to tpm library.
Signed-off-by: Praveen K Paladugu <prapal@linux.microsoft.com>
Co-authored-by: Sean Yoo <t-seanyoo@microsoft.com>
There is no need for this test any longer as we have plenty of other
tests that reboot the VM.
Further this test used unmodified bionic image, which not only will be
EOLed soon but also took a long time to shutdown as it still had snapd
installed.
Fixes: #4849
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
TEST=Boot `--disk readonly=on` along with a guest that tries to write
(unmodified hypervisor-fw) and observe that the virtio device thread no
longer panics.
Fixes: #4888
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
It's perfectly reasonable to expect if that some virtio threads trigger
libc behaviour that needs mprotect() that all virtio threads would do
the same.
Fixes: #4874
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
It is required to close all file descriptors pointing to an opened TAP
device prior to reopening the TAP device; otherwise it will return
-EBUSY as the device can only be opened once (excluding MQ use cases.)
When rebooting the VM the virtio-net threads would still be running and
so the TAP file descriptor may not have been closed. To ensure that the
TAP FD is closed wait for all the epoll threads to exit after receiving the
KILL_EVENT.
Fixes: #4868
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
If the memory is not backed by a file then it is possible to enable
Transparent Huge Pages on the memory and take advantage of the benefits
of huge pages without requiring the specific allocation of an appropriate
number of huge pages.
TEST=Boot and see that in /proc/`pidof cloud-hypervisor`/smaps that the
region is now THPeligible (and that also pages are being used.)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
An integer overflow from our virtio-mem device can be triggered
from (misbehaved) guest driver with malicious requests. This patch
handles this integer overflow explicitly and treats it as an invalid
request.
Note: this bug was detected by our virtio-mem fuzzer through 'oss-fuzz'.
Signed-off-by: Bo Chen <chen.bo@intel.com>
To support all virtio-devices, this patch replaces the customized
EpollHelper::run` with customized `EpollHelper::run_with_timeout` for
fuzzing.
Signed-off-by: Bo Chen <chen.bo@intel.com>
When running the glibc and musl integration tests on the CI after each
other skip building the kernel a second time.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This has resulted in the content being indented with spaces rather tabs
per Groovy coding style.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
As huge pages are always MAP_SHARED then where the shared memory would
be checked (for vhost-user and local migration) we can also check
instead for huge pages.
The checking is also extended to cover the memory zones based
configuration as well.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
We can't use MAP_ANONYMOUS and still have huge pages so MAP_SHARED is
effectively required when using huge pages.
Unfortunately it is not as simple as always forcing MAP_SHARED if
hugepages is on as this might be inappropriate in the backing file case
hence why there is additional complexity of assigning to mmap_flags on
each case and the MAP_SHARED is only turned on for the anonymous file
huge page case as well as anonymous shared file case.
See: #4805
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
If we do not need an anonymous file backing the memory then do not
create one.
As a side effect this addresses an issue with CoW (mmap with MAP_PRIVATE
but no MAP_ANONYMOUS) when the memory is pinned for VFIO.
Fixes: #4805
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The probing logic wasn't updated to reflect the actual opcodes in use
for io_uring which are vectored read/writes not the unvectored versions.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
After testing, io_uring_is_supported() causes about 38ms of
overhead when creating virtio-blk. By modifying the position
of io_uring_is_supported(), the overhead of creating virtio-blk
is reduced to less than 1ms when we close io_uring.
Signed-off-by: Jinrong Liang <cloudliang@tencent.com>
Since the processing of the console inputs was moved from the VMM thread
to the virtio-console thread (#3061), we have been using the 'FILE_EVENT'
to handle input from stdin/pty/file, which made 'INPUT_EVENT' obsoleted.
Signed-off-by: Bo Chen <chen.bo@intel.com>
Until there is a need for sharing the memory fd with a child process, we
should err on the safe side to close it on exec.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Following the new design proposal to improve the restore codepath when
migrating a VM, all virtio devices are supplied with an optional state
they can use to restore from. The restore() implementation every device
was providing has been removed in order to prevent from going through
the restoration twice.
Here is the list of devices now following the new restore design:
- Block (virtio-block)
- Net (virtio-net)
- Rng (virtio-rng)
- Fs (vhost-user-fs)
- Blk (vhost-user-block)
- Net (vhost-user-net)
- Pmem (virtio-pmem)
- Vsock (virtio-vsock)
- Mem (virtio-mem)
- Balloon (virtio-balloon)
- Watchdog (virtio-watchdog)
- Vdpa (vDPA)
- Console (virtio-console)
- Iommu (virtio-iommu)
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The add_device() function, from the device manager code, takes a
DeviceConfig as a parameter, instead of a VmAddDevice.
The change was originally done as part of 34412c9b41 and it didn't
break Kata Containers because the VmAddDevice and DeviceConfig structs
share most of their fields, besides the optional for serialization
`pci_segment`, which is not used by the client yet.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Considering error messages will be mostly nested, ensuring no
punctuation at the end will make the error log more readable.
Signed-off-by: Bo Chen <chen.bo@intel.com>
With known number of queues and queue events, we can make each of them
more explicit and avoid using vector/direct indexing, which is cleaner
and slightly more efficient.
Signed-off-by: Bo Chen <chen.bo@intel.com>
The the number of queues and associated events is known and fixed. We
can define and use each of them explicitly and avoid using vector (and
hence direct indexing), which is cleaner and slightly more efficient.
Signed-off-by: Bo Chen <chen.bo@intel.com>
The the number of queues and associated events is known and fixed. We
can define and use each of them explicitly and avoid using vector (and
hence direct indexing), which is cleaner and slightly more efficient.
Also, this refactoring makes it clearer that we are not handling "event
queue" events (as "_event_queue" is not being used intentionally).
Signed-off-by: Bo Chen <chen.bo@intel.com>
In this way, the virtio-iommu code can properly report an error when
a wrong number of queues is provided, instead of triggering an
out-of-bound error.
Signed-off-by: Bo Chen <chen.bo@intel.com>
This is preliminary work to ensure a migrated VM is created right before
it is restored. This will be useful when moving to a design where the VM
is both created and restored simultaneously from the Snapshot.
In details, that means the MemoryManager is the object that must be
created upon receiving the config from the source VM, so that memory
content can be later received and filled into the GuestMemory.
Only after these steps happened, the snapshot is received from the
source VM, and the actual Vm object can be created from both the
snapshot and the MemoryManager previously created.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The kernel will trigger a SIGBUS upon hugetlb page faults when there is
no huge pages available. We neither have a way to ensure enough huge
pages available on the host system, nor have a way to gracefully report
the lack of huge pages in advance from Cloud Hypervisor. For these
reasons, we have to avoid using huge pages from the virtio-mem fuzzer to
avoid SIGBUS errors.
Signed-off-by: Bo Chen <chen.bo@intel.com>
These look alarming if you are booting with the a distro kernel which is
now a recommended approach.
See: #4786
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The restore path of MemoryManager is handled specially without
implementing a `Snapshottable:restore()`. Removing the explicit call to
it along the migration code path to avoid confusions.
See: #4783
Signed-off-by: Bo Chen <chen.bo@intel.com>
The systemd journal has a known issue of generating large size logs[1],
which makes it unreliable as a source for retrieving system
information, such as for counting reboot times. This is particularly
problematic on disk-constrained systems, like the VMs we launched for
our integration tests, where the disk size is normally 2GB. By default,
the systemd journal has a size limit of 10% of the size of the
underlying file system (e.g. around 200MB for the VMs of our integration
tests), which would remove archived journal files on demand.
A better alternative to count reboot times is based on information from
`wtmp` (e.g. the login records) which is much more concise and can be
accessed via the `last` command.
[1] https://github.com/systemd/systemd/issues/5285Fixes: #4749
Signed-off-by: Bo Chen <chen.bo@intel.com>
Refresh our README in a consistent style and update it to reflect:
* A recommendation to use binaries
* Clarify our relationship with other Rust based VMMs/Rust-VMM project
* Ensure instructions result in a usable image (cloud-init)
* Simplify script instructions
* Move compilation details elsewhere
* Add Fedora 36 image details
* Point to CLOUDHV as well as Rust Hypervisor Firmware
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In function create_for_size(), when calculating the number
of clusters for the L1 table, it incorrectly divides the total number of
entries in the L1 table by the cluster size. In fact, it should be
divided by the number of entries/addresses that can fit into a cluster.
Signed-off-by: lv.mengzhao <lv.mengzhao@zte.com.cn>
Vdpa now implements the Migratable trait, which allows the device to be
added to the DeviceTree and therefore allows live migrating any vDPA
device that supports being suspended.
Given a vDPA device can't be resumed from a suspended state without
having to reset everything, we don't support pause/resume for a vDPA
device, as well as snapshot/restore (which requires resume to be
supported).
In order for the migration to work locally, reusing the same device on
the same host machine, the vhost-vdpa handler is dropped after the
snapshot has been performed, which allows the destination VM to open the
device without any conflict about the device being busy.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
In order to anticipate for migration support, we need to be able to
create a Vdpa object without VhostKernVdpa object associated with it.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Adding VHOST_VDPA_GET_CONFIG_SIZE and VHOST_VDPA_SUSPEND to the list of
authorized ioctls for the vmm thread.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
When restoring a VM, the BAR type can be found directly from the
snapshot resources. It is more reliable than the previous method which
was using self.use_64bit_bar from VirtioPciDevice because at the time
the BARs are allocated, the VirtioDevice hasn't been restored yet,
meaning the way to determine the value of use_64bit_bar is wrong for a
device like vDPA. At this time, the device type is not known and relying
on the stored resources is the only reliable way.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
With an updated Linux kernel the kernel reprograms the same table entry
in the MSI-X table multiple times. Fortunately it does so with entry
masked (the default.)
The costly part of MSI-X table reprogramming is going out to the host
kernel to update the GSI routing entries. This change makes three
optimisations.
1. If the table entry is unchanged: skip handking updates
2. Only update the GSI routing table if the table entry is unmasked
(this skips extra calls to the ioctl() for reprogramming the
entries.)
3. Only generate a message on entry unmasking if the global MSI-X enable
bit is set.
Fixes: #4273
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
In this way, we have all functions related to generate default values of
vm-config structs in the same location.
Signed-off-by: Bo Chen <chen.bo@intel.com>
These have been replaced by members of PayloadConfig and should be
removed in v28.0 (mentioned in v26.0 release notes.)
Fixes: #4737
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
This is consistent when considering that some structs have a
`#[derive(Default)`] so it makes sense for the default implementations
to be in the same location.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Place the data structures that are required for constructing a VmConfig
into it's own module from the logic that exists to suppot them.
This is useful as a consumer of the API can now clearly see what data
structures make up the API for creating VMs.
This has no functional change and I made no attempt to clean up the
ordering (it's as in the original file) nor any other clean up.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The RNG device never reads from the guest memory it reads from a file
and writes to the guest memory.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Don't silently ignore the descriptors provided by the guest. This is
consistent with other devices.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
With the guest memory size of 1MB, a valid descriptor size can be close
to the guest memory size (e.g. 1MB) and can contain close to 256k
valid pfn entries (each entry is 4 bytes). Multiplying the queue
size (e.g. 256), there can be close to 64 millions pfn entries to
process in a single request. This is why the oss-fuzz reported a
timeout (with a limit of 60s).
By reducing the guest memory size and the queue size, the worst-case now
is 8 million pfn entries for fuzzing, which can be finished in around 20
seconds according to my local experiment.
Signed-off-by: Bo Chen <chen.bo@intel.com>
This option is needed for the openapi consumer (e.g. Kata Containers) to
load firmware (e.g. td-shim) for booting.
Signed-off-by: Bo Chen <chen.bo@intel.com>
Previously, the API response was always written to stdout, but that may
not be appropriate for all clients.
The client can now control this behaviour as the client API returns the
response in the `Result`.
Fixes: #4703.
Signed-off-by: James O. D. Hunt <james.o.hunt@intel.com>
The existing API client only allows access to "VM" operations, so added
a new `simple_api_full_command_with_fds()` that allows access to "VMM"
operations too.
Also added a `simple_api_full_command()` to avoid having to specify the
file descriptors, in a similar manner to `simple_api_command()`.
Fixes: #4701.
Signed-off-by: James O. D. Hunt <james.o.hunt@intel.com>
Commit b2d1dd65f3 ("build: Remove "fwdebug" and "common" feature
flags") removed support for common features flag. But some of the
scripts are still using that parameter specifically test related to mshv
hypervisor. Thus fix those scripts by removing common feature flag.
Signed-off-by: Smit Gardhariya <sgardhariya@microsoft.com>
With the virtio-rng device the descriptors that are provided by the
guest must be writable and of non-zero length. Also propagate an error
if writing to the guest memory fails.
Signed-off-by: Bo Chen <chen.bo@intel.com>
This simplifies the CI process but also logical with the existing
functionality under "guest_debug" (dumping guest memory).
Fixes: #4679
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
TDX functionality is not currently available on MSHV but we should not
preclude building a binary that can run on both.
Fixes: #4677
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Adding the support for the user to set the MTU for the vhost-user-net
backend, which allows the integration test to be extended with the test
of the MTU parameter.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Adjust MTU logic such that:
1. Apply an MTU to the TAP interface if the user supplies it
2. Always query the TAP interface for the MTU and expose that.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This simplifes the buld and checks with very little overhead and the
fwdebug device is I/O port device on 0x402 that can be used by edk2 as a
very simple character device.
See: #4679
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
As the virt queues are initialized with random bytes from the fuzzing
engine, a descriptor buffer for the available ring can have a very large
length (e.g. up to 4GB). This means there can be up to 1 billion
entries (e.g. page frame number) for virtio-balloon to process a signal
available descriptor (given each entry is 4 bytes). This is the reason
why oss-fuzz reported a hanging issue for this fuzzer, where the
generated descriptor buffer length is 4,278,321,152.
We can avoid this kind of long execution by reducing the size of guest
memory. For example, with 1MB of guest memory, the number of descriptor
entries for processing is limited ~256K.
Signed-off-by: Bo Chen <chen.bo@intel.com>
This function is for really for the transport layer to trigger a device
reset. Instead name it appropriately for the fuzzing specific use case.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add tracing of the VM boot sequence from the point at which the request
to create a VM is received to the hand-off to the vCPU threads running.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add a new feature "tracing" that enables tracing functionality via the
"tracer" crate (sadly features and crates cannot share the same name.)
Setup: tracer::start()
The main functionality is a tracer::trace_scope()! macro that will add
trace points for the duration of the scope. Tracing events are per
thread.
Finish: tracer::end() this will write the trace file (pretty printed
JSON) to a file in the current directory.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Add a new "mtu" parameter to the NetConfig structure and therefore to
the --net option. This allows Cloud Hypervisor's users to define the
Maximum Transmission Unit (MTU) they want to use for the network
interface that they create.
In details, there are two main aspects. On the one hand, the TAP
interface is created with the proper MTU if it is provided. And on the
other hand the guest is made aware of the MTU through the VIRTIO
configuration. That means the MTU is properly set on both the TAP on the
host and the network interface in the guest.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This was unfortunately missing from
430bfd38be which disabled the other builds
that rely on the bare metal systems.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
There's no need to delegate the resize operation to the virtio-mem
thread. This can come directly from the vmm thread which will use the
Mem object to update the VIRTIO configuration and trigger the interrupt
for the guest to be notified.
In order to achieve what's described above, the VirtioMemZone structure
now has a handle onto the Mem object directly. This avoids the need for
intermediate Resize and ResizeSender structures.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The fuzzer exercises the inflate, deflate and reporting events of
virtio-balloon via creating three queues and kicking three events.
Signed-off-by: Bo Chen <chen.bo@intel.com>
To make the fuzzer faster and more effective, the guest memory is
setup with a much smaller size (comparing with other virtio device
fuzzers) and a hole between the memory for holding virtio queue and
the rest of guest data. It brings two benefits: 1) avoid writing large
chunk of data from 'urandom' into the available descriptor chain (which
makes the fuzzer faster); 2) reduce substantial amount of overwrites to
the virtio queue data by the data from 'urandom (which makes the fuzzer
more deterministic and hence effective).
Signed-off-by: Bo Chen <chen.bo@intel.com>
With the existing code, `round_up(7, 2)` would generate `6` which is
obviously wrong. Also, following what's done for 'round_down()', the
fixed code does not handle 'alignment == 0' explicitly.
Signed-off-by: Bo Chen <chen.bo@intel.com>
With the existing macro, `align!(7, 4)` would generate `16` which is
obviously wrong. Also, given it is a macro, the compiler catch the error
if the provided 'alignment' is '0'.
Signed-off-by: Bo Chen <chen.bo@intel.com>
There's no need to delegate the resize operation to the virtio-balloon
thread. This can come directly from the vmm thread which will use the
Balloon object to update the VIRTIO configuration and trigger the
interrupt for the guest to be notified.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Given the AMX x86 feature has been made available since kernel v5.17,
and given we don't have any test validating this feature, there's no
need to keep it behing a Rust feature gate.
Fixes#3996
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This makes it consistent with x86_64 and allows the performance metrics
tooling to continue to work on aarch64.
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Update the implementation of the process_queue() function to match all
other virtio devices implementations. This solves some issue related to
potential out-of-bound accesses to the former used_desc_heads list.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This reduces the binary size from 7.3M to 6.5M.
Also the build time for a clean build shrinks from 1m28s to 1m14s on my
development machine (an eight-core VM on Azure). This goes against the
conventional wisdom but it is what it is.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
The latest kvm-ioctls contains a breaking change to its API. Now Arm's
get/set_one_reg use u128 instead of u64.
Signed-off-by: Wei Liu <liuwe@microsoft.com>
Packages targeting various distributions are now available thanks to the
Open Bulid Service. The SPECS have been moved to a separate repo under
https://github.com/cloud-hypervisor/obs-packaging
which documents how to include pre-compiled binaires in a supported
distribution. Besides the Cloud Hypervisor itself, accompanying pieces
like UEFI bins are available, too.
Any contributions to the packaging can be also accepted in
the aforementioned repo. Further integration with the OBS like PR
validation is to be targeted there, too, as far as it can get. As the
packaging topic has moved in a separate repo, the bundled SPEC became
obsolete is suggested for removal.
Fixes: #4257
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
Co-authored-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
The fuzzer is focusing on the virtio-pmem code that processes guest
inputs (e.g. virt queues). Given 'flush' is the only virtio-pmem
request, the fuzzer is essentially testing the code for parsing and
error handling.
Signed-off-by: Bo Chen <chen.bo@intel.com>
To make the fuzzers more focused and more efficient, we now provide
default addresses for the descriptor table, available ring, and used
ring, which ensures the virt-queue has a valid memory layout (e.g. no
overlapping between descriptor tables, available ring, and used ring).
Signed-off-by: Bo Chen <chen.bo@intel.com>
Given rate limiter tests are performance related, they need to be tested
on a bare-metal machine. It is now using the same node as the metrics CI.
Signed-off-by: Bo Chen <chen.bo@intel.com>
The helper functions for measuring and parsing the performance of
virtio-net and virtio-block devices are moved to the `test_infra` crate
so that they be reused for integration tests of rate limiter.
Signed-off-by: Bo Chen <chen.bo@intel.com>
Multiple rust-vmm crates must be updated at once given the vm-memory one
has been updated and they all rely on vm-memory.
- vm-memory from 0.8.0 to 0.9.0
- vhost from 0.4.0 to 0.5.0
- virtio-queue from 0.5.0 to 0.6.0
- vhost-user-backend from 0.6.0 to 0.7.0
- linux-loader from 0.4.0 to 0.5.0
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
This function starts the 'receive-migration' for the destination VM,
'send-migration' for the source VM, waits for the live-migration
completion, and prints debug information upon errors.
Signed-off-by: Bo Chen <chen.bo@intel.com>
This patch moves the actual test logic and assertions from various
functions to the actual tests, which makes these tests more readable and
easier to debug.
Signed-off-by: Bo Chen <chen.bo@intel.com>
The baremetal CI for x86-64 was disabled while the machines were out for
maintenance. Now that the machines are back up, let's re-enable the full
CI.
This reverts commit 476e30f5cb.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Until the issue #4583 is resolved, we must disable this test given it's
failing quite often on the aarch64 worker.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Removing the option --tdx to specify that we want to run a TD VM. Rely
on --platform option by adding the "tdx" boolean parameter. This is the
new way for enabling TDX with Cloud Hypervisor.
Along with this change, the way to retrieve the firmware path has been
updated to rely on the recently introduced PayloadConfig structure.
Fixes#4556
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Rather than overloading the GED device with the detection of what the
underlying event means move that to a new generic event controller. The
advantage of this is that that the generic event controller can have a
_CRS that covers the memory for the OperationRegion (the ged driver
rejects _CRS entries that contain anything beyond the single interrupt)
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
The PCI buses should not declare the address space related to the MMIO
config space given it's already declared in the MCFG table and through
the motherboard device PNP0C02 in the DSDT table.
The PCI MMIO config region for the segment was being wrongly exposed as
part of the _CRS for the ACPI bus device (using Memory32Fixed). Exposing
it via this object was ineffectual as the equivalent entry in the
PNP0C02 (_SB_.MBRD) marked those ranges as not usable via the kernel.
Either way, with both devices used by the kernel, the kernel will not
try and use those memory ranges for the device BARs. However under
td-shim on TDX the PNP0C02 device is not on the permitted list of
devices so the the memory ranges were not marked as unusable resulting
in the kernel attempting to allocate BARs that collided with the PCI
MMIO configuration space.
This is based on the kernel documentation PCI/acpi-info.rst which relies
on ACPI and PCI Firmware specifications. And here are the interesting
quotes from this document:
"""
Prior to the addition of Extended Address Space descriptors, the failure
of Consumer/Producer meant there was no way to describe bridge registers
in the PNP0A03/PNP0A08 device itself. The workaround was to describe the
bridge registers (including ECAM space) in PNP0C02 catch-all devices.
With the exception of ECAM, the bridge register space is device-specific
anyway, so the generic PNP0A03/PNP0A08 driver (pci_root.c) has no need
to know about it.
PNP0C02 “motherboard” devices are basically a catch-all. There’s no
programming model for them other than “don’t use these resources for
anything else.” So a PNP0C02 _CRS should claim any address space that is
(1) not claimed by _CRS under any other device object in the ACPI
namespace and (2) should not be assigned by the OS to something else.
The address range reported in the MCFG table or by _CBA method (see
Section 4.1.3) must be reserved by declaring a motherboard resource. For
most systems, the motherboard resource would appear at the root of the
ACPI namespace (under _SB) in a node with a _HID of EISAID (PNP0C02),
and the resources in this case should not be claimed in the root PCI
bus’s _CRS. The resources can optionally be returned in Int15 E820 or
EFIGetMemoryMap as reserved memory but must always be reported through
ACPI as a motherboard resource.
"""
This change has been manually tested by running a VM with multiple
segments (4 segments), and by hotplugging an additional disk to the
segment number 2 (third segment).
From one shell:
"""
cloud-hypervisor \
--cpus boot=1 \
--memory size=1G \
--kernel vmlinux \
--cmdline "root=/dev/vda1 rw console=hvc0" \
--disk path=jammy-server-cloudimg.raw \
--api-socket /tmp/ch.sock \
--platform num_pci_segments=4
"""
From another shell (after the VM is booted):
"""
ch-remote \
--api-socket=/tmp/ch.sock \
add-disk \
path=test-disk.raw,id=disk2,pci_segment=2
"""
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
As 'handle_child_output()' may terminate the test on panic, we need to
cleanup ovs-dpdk setup in advance.
see: #4555
Signed-off-by: Bo Chen <chen.bo@intel.com>
Instead of always fuzzing virt-queues with default values (mostly 0s),
the fuzzer now initializes the virt-queue based on the fuzzed input
bytes, such as the tail position of the available ring, queue size
selected by driver, descriptor table address, available ring address,
used ring address, etc. In this way, the fuzzer can explore the
virtio-block code path with various virt-queue setup.
Signed-off-by: Bo Chen <chen.bo@intel.com>
Use VgicConfig to initialize Vgic.
Use Gic::create_default_config everywhere so we don't always recompute
redist/msi registers.
Add a helper create_test_vgic_config for tests in hypervisor crate.
Signed-off-by: Nuno Das Neves <nudasnev@microsoft.com>
VgicConfig structure will be used for initializing the Vgic.
Gic::create_default_config will be used everywhere we currently compute
redist/msi registers.
Signed-off-by: Nuno Das Neves <nudasnev@microsoft.com>
AArch64 can share the same way of loading payload with x86_64. It makes
the payload loading more consistent between different arches.
Signed-off-by: Jianyong Wu <jianyong.wu@arm.com>
uefi_flash is used when load firmware, that is load payload depends on
device manager. move uefi_flash to memory manager can eliminate the
dependency.
Signed-off-by: Jianyong Wu <jianyong.wu@arm.com>
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
A new firmware item has been added into payload config, we need
extend ability to load standalone firmware on AArch64.
"load_kernel" method will be the entry of image loading work including
kernel and firmware.
This change is back compatible. So, we can either load firmware through
"-kernel" like before or "-firmware".
Signed-off-by: Jianyong Wu <jianyong.wu@arm.com>
Later, we will load standalone firmware. So, refactor load_kernel
by abstracting load_firmware method.
Signed-off-by: Jianyong Wu <jianyong.wu@arm.com>
It provides fuzzer a reliable way to wait for a sequence of events
to complete for virtio-devices while not using a fixed timeout to
maintain the full speed of fuzzing.
Take virtio-block as an example, the 'queue event' with a valid
available queue setup can trigger a 'completion event'. This is a
meaningful virtio-block code path of processing guest inputs which is
our target for fuzzing virtio devices.
Signed-off-by: Bo Chen <chen.bo@intel.com>
Fetch the whole git repository (not just the specific commit) and use
the github context instead of hardcoded branch.
Unfortunately now that we process the list of revisions correctly it
shows that the checks don't work on aarch64 due to cross limitations so
this has been removed.
Fixes: #4523
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
Given the virtio-console is now able to buffer its output when no PTY is
connected on the other end, the device manager code is updated to enable
this. Moving the endpoint type from FilePair to PtyPair enables the
proper codepath in the virtio-console implementation, as well as
updating the PTY resize code, and forcing the PTY to always be
non-blocking.
The non-blocking behavior is required to avoid blocking the guest that
would be waiting on the virtio-console driver. When receiving an
EWOULDBLOCK error, the output will simply be redirected to the temporary
buffer so that it can be later flushed.
The PTY resize logic has been slightly modified to ensure the PTY file
descriptors are closed. It avoids the child process to keep a hold onto
the PTY device, which would have caused the PTY to believe something is
connected on the other end, which would have prevented the detection of
any new connection on the PTY.
Fixes#4521
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Through multiple changes, this patch aims at providing a reliable
solution for detecting the state of the PTY's connection. Being able to
find out when the other end of the PTY is connected is essential to
prevent the loss of data being output through the PTY. When the PTY
isn't connected, the output is buffered through the SerialBuffer, the
same solution that was created for the serial port initially.
Fixes#4521
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Extending and improving both the structure and the trait allows for more
flexibility regarding what can be achieved with the epoll loop. It
allows for a timeout to be configured instead of the default blocking
behavior. There is a new method in the trait to notify the caller that
the timeout has been reached. And there's a new knob to be notified with
the full list of events before the internal code will actually loop over
every event.
All of these new features are not affecting the previous behavior, and
using EpollHelper::run() should be unchanged.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We want to be able to reuse the SerialBuffer from the virtio-devices
crate, particularly from the virtio-console implementation. That's why
we move the SerialBuffer definition to its own crate so that it can be
accessed from both vmm and virtio-devices crates, without creating any
cyclic dependency.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
If the epoll_wait() call returns EINTR, this only means a signal has
been delivered before any of the file descriptors registered triggered
an event or before the end of the timeout (if timeout isn't -1). For
that reason, we should simply try to listen on the epoll loop again.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
We must limit how much the buffer can grow, otherwise this could lead
the process to consume all the memory on the machine. This could happen
if the output from the guest was very important and nothing would
connect to the PTY for a long time.
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Set the maximum number of HW breakpoints according to the value returned
from `Hypervisor::get_guest_debug_hw_bps()`.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
Added `Hypervisor::get_guest_debug_hw_bps()` for fetching the number of
supported hardware breakpoints.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
On AArch64, `translate_gva` API is not provided by KVM. We implemented
it in VMM by walking through translation tables.
Address translation is big topic, here we only focus the scenario that
happens in VMM while debugging kernel. This `translate_gva`
implementation is restricted to:
- Exception Level 1
- Translate high address range only (kernel space)
This implementation supports following Arm-v8a features related to
address translation:
- FEAT_LPA
- FEAT_LVA
- FEAT_LPA2
The implementation supports page sizes of 4KiB, 16KiB and 64KiB.
Signed-off-by: Michael Zhao <michael.zhao@arm.com>
The goal of this patch is to provide a reliable way to detect when the
other end of the PTY is connected, and therefore be able to identify
when we can write to the PTY device. This is needed because writing to
the PTY device when the other end isn't connected causes the loss of
the written bytes.
The way to detect the connection on the other end of the PTY is by
knowing the other end is disconnected at first with the presence of the
EPOLLHUP event. Later on, when the connection happens, EPOLLHUP is not
triggered anymore, and that's when we can assume it's okay to write to
the PTY main device.
It's important to note we had to ensure the file descriptor for the
other end was closed, otherwise we would have never seen the EPOLLHUP
event. And we did so by removing the "sub" field from the PtyPair
structure as it was keeping the associated File opened.
Fixes#3170
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
Reads from the random file may only be partial, e.g., if the random file is an ordinary text
file. When that happens, the device needs to signal to the driver that only parts of the buffer have
been overwritten.
Signed-off-by: Markus Napierkowski <markus.napierkowski@cyberus-technology.de>
Following our recent v26.0 release we can re-enable our live upgrade
tests to try and make it possible for us to move to making LTS releases.
Currently limited to x86-64 as the live upgrade tests fail on aarch64.
Fixes: #3949
Signed-off-by: Rob Bradford <robert.bradford@intel.com>
sh 'scripts/dev_cli.sh tests --metrics -- -- --report-file /root/workloads/metrics.json'
}
}
stage ('Upload metrics report') {
steps {
sh 'curl -X PUT https://cloud-hypervisor-metrics.azurewebsites.net/api/publishmetrics -H "x-functions-key: $METRICS_PUBLISH_KEY" -T ~/workloads/metrics.json'
}
}
}
}
}
}
}
post {
regression {
script {
if (env.BRANCH_NAME == 'main') {
slackSend (color: '#ff0000', message: '"main" branch build is now failing')
}
}
}
fixed {
script {
if (env.BRANCH_NAME == 'main') {
slackSend (color: '#00ff00', message: '"main" branch build is now fixed')
sh 'scripts/dev_cli.sh tests --metrics -- -- --report-file /root/workloads/metrics.json'
}
}
stage('Upload metrics report') {
steps {
sh 'curl -X PUT https://cloud-hypervisor-metrics.azurewebsites.net/api/publishmetrics -H "x-functions-key: $METRICS_PUBLISH_KEY" -T ~/workloads/metrics.json'
}
}
}
}
stage('Worker build - Rate Limiter') {
agent { node { label 'focal-metrics' } }
when {
branch 'main'
beforeAgent true
expression {
return runWorkers
}
}
stages {
stage('Checkout') {
steps {
checkout scm
}
}
stage('Run rate-limiter integration tests') {
options {
timeout(time: 10, unit: 'MINUTES')
}
steps {
sh 'scripts/dev_cli.sh tests --integration-rate-limiter'
}
}
}
}
stage('Worker build - SGX') {
agent { node { label 'jammy-sgx' } }
when {
beforeAgent true
allOf {
branch 'main'
expression {
return runWorkers
}
}
}
stages {
stage('Checkout') {
steps {
checkout scm
}
}
stage('Run SGX integration tests') {
options {
timeout(time: 1, unit: 'HOURS')
}
steps {
sh 'scripts/dev_cli.sh tests --integration-sgx'
}
}
stage('Run SGX integration tests for musl') {
options {
timeout(time: 1, unit: 'HOURS')
}
steps {
sh 'scripts/dev_cli.sh tests --integration-sgx --libc musl'
}
}
}
post {
always {
sh "sudo chown -R jenkins.jenkins ${WORKSPACE}"
deleteDir()
}
}
}
stage('Worker build - VFIO') {
agent { node { label 'jammy-vfio' } }
when {
beforeAgent true
allOf {
branch 'main'
expression {
return runWorkers
}
}
}
stages {
stage('Checkout') {
steps {
checkout scm
}
}
stage('Run VFIO integration tests') {
options {
timeout(time: 1, unit: 'HOURS')
}
steps {
sh 'scripts/dev_cli.sh tests --integration-vfio'
}
}
stage('Run VFIO integration tests for musl') {
options {
timeout(time: 1, unit: 'HOURS')
}
steps {
sh 'scripts/dev_cli.sh tests --integration-vfio --libc musl'
}
}
}
post {
always {
sh "sudo chown -R jenkins.jenkins ${WORKSPACE}"
deleteDir()
}
}
}
}
}
}
post {
regression {
script {
if (env.BRANCH_NAME == 'main') {
slackSend(color: '#ff0000', message: '"main" branch build is now failing', channel: '#jenkins-ci')
}
}
}
fixed {
script {
if (env.BRANCH_NAME == 'main') {
slackSend(color: '#00ff00', message: '"main" branch build is now fixed', channel: '#jenkins-ci')
}
}
}
}
}
def cancelPreviousBuilds() {
// Check for other instances of this particular build, cancel any that are older than the current one
Cloud Hypervisor also supports direct kernel boot into a `vmlinux` ELF kernel.
In order to support virtio-watchdog we have our own development branch. You are
of course able to use your own kernel but these instructions will continue with
the version that we develop and test against.
#### Building your Kernel
Cloud Hypervisor also supports direct kernel boot. For x86-64, a `vmlinux` ELF kernel (compiled with PVH support) is needed. In order to support development there is a custom branch; however provided the required options are enabled any recent kernel will suffice.
are all equal and welcome means of contribution. See the
[CONTRIBUTING](CONTRIBUTING.md) document for more details.
## Join us
## Slack
Get an [invite to our Slack channel](https://join.slack.com/t/cloud-hypervisor/shared_invite/enQtNjY3MTE3MDkwNDQ4LWQ1MTA1ZDVmODkwMWQ1MTRhYzk4ZGNlN2UwNTI3ZmFlODU0OTcwOWZjMTkwZDExYWE3YjFmNzgzY2FmNDAyMjI)
and [join us on Slack](https://cloud-hypervisor.slack.com/).
## Mailing list
Please report bugs using the [GitHub issue
tracker](https://github.com/cloud-hypervisor/cloud-hypervisor/issues) but for
broader community discussions you may use our [mailing
list](https://lists.cloudhypervisor.org/g/dev/).
## Security issues
Please contact the maintainers listed in the MAINTAINERS.md file with security issues.
- [Containerized builds and tests](#containerized-builds-and-tests)
# Building Cloud Hypervisor
We recommend users use the pre-built binaries that are mentioned in the README.md file in the root of the repository. Building from source is only necessary if you wish to make modifications.
## Preparation
We create a folder to build and run `cloud-hypervisor` at `$HOME/cloud-hypervisor`
```shell
$ exportCLOUDH=$HOME/cloud-hypervisor
$ mkdir $CLOUDH
```
## Install prerequisites
You need to install some prerequisite packages in order to build and test Cloud
Hypervisor. Here, all the steps are based on Ubuntu, for other Linux
distributions please replace the package manager and package name.
```shell
# Install basic packages needed. For a package list targeting for more
# functionalities for example the test, please see resources/Dockerfile.
@@ -4,7 +4,7 @@ Currently Cloud Hypervisor supports hot plugging of CPUs devices (x86 only), PCI
## Kernel support
For hotplug on Cloud Hypervisor ACPI GED support is needed. This can either be achieved by turning on `CONFIG_ACPI_REDUCED_HARDWARE_ONLY`
For hotplug on Cloud Hypervisor ACPI GED support is needed. This can either be achieved by turning on `CONFIG_ACPI_REDUCED_HARDWARE_ONLY`
or by using this kernel patch (available in 5.5-rc1 and later): https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/patch/drivers/acpi/Makefile?id=ac36d37e943635fc072e9d4f47e40a48fbcdb3f0
The extra vCPU threads will be created and advertised to the running kernel. The kernel does not bring up the CPUs immediately and instead the user must "online" them from inside the VM:
@@ -56,7 +56,7 @@ After a reboot the added CPUs will remain.
Removing CPUs works similarly by reducing the number in the "desired_vcpus" field of the reisze API. The CPUs will be automatically offlined inside the guest so there is no need to run any commands inside the guest:
The new memory is now available to use inside the VM:
@@ -159,7 +159,7 @@ The same API can also be used to reduce the desired RAM for a VM. It is importan
Extra PCI devices can be added and removed from a running `cloud-hypervisor` instance. This is controlled by making a HTTP API request to the VMM to ask for the additional device to be added, or for the existing device to be removed.
Note: On AArch64 platform, PCI device hotplug can only be achieved using ACPI. Please refer to the [documentation](arm64.md#uefi-booting) for more information.
Note: On AArch64 platform, PCI device hotplug can only be achieved using ACPI. Please refer to the [documentation](uefi.md#building-uefi-firmware-for-aarch64) for more information.
To use PCI device hotplug start the VM with the HTTP server.
@@ -244,7 +244,7 @@ After a reboot the added PCI device will remain.
Removing a PCI device works the same way for all kind of PCI devices. The unique identifier related to the device must be provided. This identifier can be provided by the user when adding the new device, or by default Cloud Hypervisor will assign one.
> The latest version of TDShim being tested is [_66bb334_](https://github.com/confidential-containers/td-shim/tree/66bb33451befbf1291abe3cfea7ee9e99d922b0d).
This is a lightweight version of the TDVF, written in Rust and designed for
direct kernel boot, which is useful for containers use cases.
You can find the instructions for building the firmware directly from the
@@ -38,6 +38,6 @@ This level is for the benefit of developers. It should be used for sporadic and
### `debug!()`
Use `-vv` to enable.
Use `-v -v` to enable.
For the most verbose of logging messages. It is acceptable to "spam" the log with repeated invocations of the same message. This level of logging would be combined with `--log-file`.
For the most verbose of logging messages. It is acceptable to "spam" the log with repeated invocations of the same message. This level of logging would be combined with `--log-file`.
Cloud Hypervisor has a basic tracing infrastucture, particularly focussed on
the tracing of the initial VM setup.
## Usage
To enabling tracing, build with "tracing" feature. When compiled without the
feature the tracing is compiled out.
```bash
cargo build --features "tracing"
```
And then run Cloud Hypervisor as you wish, the trace will be written to the current directory as `cloud-hypervisor-<pid>.trace`. This is JSON file which you can inspect yourself.
Alternatively you can use the provided script in
`scripts/ch-trace-visualiser.py` to generate an SVG:
Cloud Hypervisor supports UEFI boot through the utilization of the EDK II based UEFI firmware.
## Building UEFI Firmware
## Building UEFI Firmware for x86-64
To avoid any unnecessary issues, it is recommended to use Ubuntu 18.04 and its default toolset. Any other compatible Linux distribution is otherwise suitable, however it is suggested to use a temporary Docker container with Ubuntu 18.04 for a quick build on an existing Linux machine.
Please note that nasm-2.15 is required for the build to succeed.
The commands below will compile an OVMF firmware suitable for Cloud Hypervisor.
Any UEFI capable image can be booted using the Cloud Hypervisor specific firmware. Windows guests under Cloud Hypervisor only support UEFI boot, therefore OVMF is mandatory there.
To make Cloud Hypervisor use UEFI boot, pass the `CLOUDHV.fd` file path as an argument to the `--kernel` option. The firmware file will be opened in read only mode.
To make Cloud Hypervisor use UEFI boot, pass the `CLOUDHV.fd` (for x86-64) / `CLOUDHV_EFI.fd` (for AArch64) file path as an argument to the `--kernel` option. The firmware file will be opened in read only mode.
VSOCK provides a way for guest and host to communicate through a socket. VSOCK sockets support both stream and datagram types.
The `virtio-vsock` is based on the [Firecracker](https://github.com/firecracker-microvm/firecracker/blob/main/docs/vsock.md) implementation, where additional details can be found.
## What is a CID?
CID is a 32-bit context identifier describing the source or destination. In combination with the port, the complete addressing can be achieved to describe multiple listeners running on the same machine.
The table below depicts the well known CID values:
| CID | Description |
|-----|-------------|
| -1 | Random CID |
| 0 | Hypervisor |
| 1 | Loopback |
| 2 | Host |
## Prerequisites
### Kernel Requirements
Host kernel: CONFIG_VHOST_VSOCK
Guest kernel: CONFIG_VIRTIO_VSOCKETS
### Nested VM support
Linux __v5.5__ or newer is required for the L1 VM.
### Loopback support
Linux __v5.6__ or newer is required.
## Establishing VSOCK Connection
VSOCK device becomes available with `--vsock` option passed by the VM start. Cloud Hypervisor can be invoked for instance as below:
```bash
cloud-hypervisor \
--cpus boot=1\
--memory size=4G \
--firmware CLOUDHV.fd \
--disk path=jammy-server-cloudimg.raw \
--vsock cid=3,socket=/tmp/ch.vsock
```
The examples use __socat__`>=1.7.4` to illustrate the VSOCK functionality. However, there are other tools supporting VSOCK, like [ncat](https://stefano-garzarella.github.io/posts/2019-11-08-kvmforum-2019-vsock/).
### Connecting from Host to Guest
The host starts to listen on the defined port:
`$ socat - VSOCK-LISTEN:1234`
Once the host is listening, the guest can send data:
`echo -e "CONNECT 1234\\nHello from host!" | socat - UNIX-CONNECT:/tmp/ch.vsock
Note the string `CONNECT <port>` prepended to the actual data. It is possible for the guest to start listening on different ports, thus the specific command is needed to instruct VSOCK to which listener the host wants to connect. It needs to be sent once per connection. Once the connection established, data transfers can take place directly.
### Connecting from Guest to Host
This first requires a listening UNIX socket on the host side. The UNIX socket path has to be constructed by using the socket path used at the VM launch time with appended `_` and the port number to be used on the guest side. As in the example above, if we'd intended to connect from the guest to the port `1234`, the Unix socket path on the host side would be `/tmp/ch.vsock_1234`.
Also note that the CID used on the guest side is the well known CID value `2`.
Listening on the host side:
`$ socat - UNIX-LISTEN:/tmp/ch.vsock_1234`
From the guest:
`$ echo -e "Hello from guest!" | socat - VSOCK-CONNECT:2:1234`
## Links
- [virtio-vsock in QEMU, Firecracker and Linux: Status, Performance and Challenges](https://kvmforum2019.sched.com/event/TmwK)
- [Leveraging virtio-vsock in the cloud and containers](https://archive.fosdem.org/2021/schedule/event/vai_virtio_vsock/)
- [VSOCK man page](https://manpages.ubuntu.com/manpages/focal/man7/vsock.7.html)
@@ -131,7 +131,7 @@ See also the [links](#Links) section for a more extended SAC documentation.
## Network
This section illustrates the Windows specific corner points for the VM network configuration. For the extended networking guide, including bridging for multiple VMs, follow [networking.md](networking.md).
This section illustrates the Windows specific aspects of the VM network configuration.
// This array is only accessed as parts of a u32 word, so interpret it as a u8 array.
// Cursors are only readable on arrays of u8, not i8(c_char).
@@ -290,6 +291,7 @@ impl LapicState {
usestd::io::Cursor;
usestd::mem;
// SAFETY: plain old data type
letsliceu8=unsafe{
// This array is only accessed as parts of a u32 word, so interpret it as a u8 array.
// Cursors are only readable on arrays of u8, not i8(c_char).
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
