# CPU

Cloud Hypervisor has many options when it comes to the creation of virtual
CPUs. This document aims to explain what Cloud Hypervisor is capable of and
how it can be used to meet the needs of very different use cases.

## Options

`CpusConfig` or what is known as `--cpus` from the CLI perspective is the way
to set vCPUs options for Cloud Hypervisor.

```rust
struct CpusConfig {
    boot_vcpus: u32,
    max_vcpus: u32,
    topology: Option<CpuTopology>,
    kvm_hyperv: bool,
    max_phys_bits: u8,
    affinity: Option<Vec<CpuAffinity>>,
    features: CpuFeatures,
    nested: bool,
    core_scheduling: CoreScheduling,
}
```

```
--cpus boot=<boot_vcpus>,max=<max_vcpus>,topology=<threads_per_core>:<cores_per_die>:<dies_per_package>:<packages>,kvm_hyperv=on|off,max_phys_bits=<maximum_number_of_physical_bits>,affinity=<list_of_vcpus_with_their_associated_cpuset>,features=<list_of_features_to_enable>,nested=on|off,core_scheduling=vm|vcpu|off
```

### `boot`

Number of vCPUs present at boot time.

This option allows defining a specific number of vCPUs to be present at the
time the VM is started. This option is mandatory when using the `--cpus`
parameter. If `--cpus` is not specified, this option takes the default value
of `1`, starting the VM with a single vCPU.

Value is an unsigned integer of 32 bits.

_Example_

```
--cpus boot=2
```

### `max`

Maximum number of vCPUs.

This option defines the maximum number of vCPUs that can be assigned to the VM.
In particular, this option is used when looking for CPU hotplug as it provides
an indication about how many vCPUs might be needed later during the runtime of
the VM.
For instance, if booting the VM with 2 vCPUs and a maximum of 6 vCPUs, it means
up to 4 vCPUs can be added later at runtime by resizing the VM.

The value must be greater than or equal to the number of boot vCPUs.
The value is an unsigned integer of 32 bits.

By default this option takes the value of `boot`, meaning vCPU hotplug is not
expected and can't be performed.

_Example_

```
--cpus max=3
```

### `topology`

Topology of the guest platform.

This option gives the user a way to describe the exact topology that should be
exposed to the guest. It can be useful to describe to the guest the same
topology found on the host as it allows for proper usage of the resources and
is a way to achieve better performance.

The topology is described through the following structure:

```rust
struct CpuTopology {
    threads_per_core: u16,
    cores_per_die: u16,
    dies_per_package: u16,
    packages: u16,
}
```

or the following syntax through the CLI:

```
topology=<threads_per_core>:<cores_per_die>:<dies_per_package>:<packages>
```

By default the topology will be `1:1:1:1`.

_Example_

```
--cpus boot=2,topology=1:1:2:1
```

### `kvm_hyperv`

Enable KVM Hyper-V emulation.

When turned on, this option relies on KVM to emulate the synthetic interrupt
controller (SynIC) along with synthetic timers expected by a Windows guest.
A Windows guest usually runs on top of Microsoft Hyper-V, therefore expects
these synthetic devices to be present. That's why KVM provides a way to emulate
them and avoids failures running a Windows guest with Cloud Hypervisor.

By default this option is turned off.

_Example_

```
--cpus kvm_hyperv=on
```

### `max_phys_bits`

Maximum size for guest's addressable space.

This option defines the maximum number of physical bits for all vCPUs, which
sets a limit for the size of the guest's addressable space. This is mainly
useful for debugging purposes.

The value is an unsigned integer of 8 bits.

_Example_

```
--cpus max_phys_bits=40
```

### `affinity`

Affinity of each vCPU.

This option gives the user a way to provide the host CPU set associated with
each vCPU. It is useful for achieving CPU pinning, ensuring multiple VMs won't
affect the performance of each other. It might also be used in the context of
NUMA as it is a way of making sure the VM can run on a specific host NUMA node.
In general, this option is used to increase the performance of a VM depending
on the host platform and the type of workload running in the guest.

The affinity is described through the following structure:

```rust
struct CpuAffinity {
    vcpu: u32,
    host_cpus: Vec<usize>,
}
```

or the following syntax through the CLI:

```
affinity=[<vcpu_id1>@[<host_cpu_id1>, <host_cpu_id2>], <vcpu_id2>@[<host_cpu_id3>, <host_cpu_id4>]]
```

The outer brackets define the list of vCPUs. And for each vCPU, the inner
brackets attached to `@` define the list of host CPUs the vCPU is allowed to
run onto.

Multiple values can be provided to define each list. Each value is a
platform-native unsigned integer (`usize`).

For instance, if one needs to run vCPU 0 on host CPUs from 0 to 4, the syntax
using `-` will help define a contiguous range with `affinity=0@[0-4]`. The
same example could also be described with `affinity=0@[0,1,2,3,4]`.

A combination of both `-` and `,` separators is useful when one might need to
describe a list containing host CPUs from 0 to 99 and the host CPU 255, as it
could simply be described with `affinity=0@[0-99,255]`.

As soon as one tries to describe a list of values, `[` and `]` must be used to
demarcate the list.

By default each vCPU runs on the entire host CPU set.

_Example_

```
--cpus boot=3,affinity=[0@[2,3],1@[0,1]]
```

In this example, assuming the host has 4 CPUs, vCPU 0 will run exclusively on
host CPUs 2 and 3, while vCPU 1 will run exclusively on host CPUs 0 and 1.
Because nothing is defined for vCPU 2, it can run on any of the 4 host CPUs.

### `features`

Set of CPU features to enable.

This option allows the user to enable a set of CPU features that are disabled
by default otherwise.

The currently available feature set is: `amx`.

The `amx` feature will enable the x86 extension adding hardware units for
matrix operations (int and float dot products). The goal of the extension is to
provide performance enhancements for these common operations.

_Example_

```
--cpus features=amx
```

In this example the amx CPU feature will be enabled for the VMM.


### `nested`

Enable nested virtualization (default on). Nested virtualization is needed to access hardware virtualization by this guest. This option can only be changed on x86-64.

_Example_

```
--cpus nested=on
```

### `core_scheduling`

Core scheduling mode for vCPU threads.

This option controls Linux core scheduling (`PR_SCHED_CORE`) for vCPU threads,
which prevents untrusted tasks from sharing SMT siblings. This mitigates
side-channel attacks (e.g. MDS, L1TF) between vCPU threads.

Three modes are available:

- `vm` (default): All vCPU threads share a single core scheduling cookie.
  vCPUs may be co-scheduled on SMT siblings of the same core, providing
  better performance while still isolating VM threads from host tasks.
- `vcpu`: Each vCPU thread gets its own unique cookie. No two vCPUs can
  share SMT siblings, providing the strongest isolation between vCPUs at
  the cost of performance.
- `off`: No core scheduling is applied.

On kernels older than 5.14 (which lack `PR_SCHED_CORE` support), the
option silently has no effect.

_Example_

```
--cpus boot=2,core_scheduling=vm
```

In this example, both vCPUs will share the same core scheduling cookie,
allowing them to be co-scheduled on SMT siblings while preventing host
threads from sharing those siblings.
