diff --git a/README.md b/README.md index fdb18255f..80048ab32 100644 --- a/README.md +++ b/README.md @@ -111,19 +111,25 @@ do not wish to use the pre-built binaries. ## Booting Linux -Cloud Hypervisor supports direct kernel boot (the x86-64 kernel requires the kernel -built with PVH support or a bzImage) or booting via a firmware (either [Rust Hypervisor -Firmware](https://github.com/cloud-hypervisor/rust-hypervisor-firmware) or an -edk2 UEFI firmware called `CLOUDHV` / `CLOUDHV_EFI`.) +Cloud Hypervisor boots guests in one of two ways. The first is direct +kernel boot, where a kernel image is passed to `--kernel`. The x86-64 +kernel must be built with PVH support or be a bzImage. The second is +firmware boot, where a firmware image is passed to `--firmware` and +brings up the guest's normal boot loader. -Binary builds of the firmware files are available for the latest release of -[Rust Hypervisor +Two firmware options are supported, and which one works best depends +on the guest OS. [Rust Hypervisor +Firmware](https://github.com/cloud-hypervisor/rust-hypervisor-firmware) +is a lightweight Rust-based PVH firmware. The edk2 UEFI firmware is +called `CLOUDHV.fd` for x86-64 and `CLOUDHV_EFI.fd` for AArch64. +Prebuilt binaries for both are available at their respective releases +pages, [Rust Hypervisor Firmware](https://github.com/cloud-hypervisor/rust-hypervisor-firmware/releases/latest) and [our edk2 -repository](https://github.com/cloud-hypervisor/edk2/releases/latest) - -The choice of firmware depends on your guest OS choice; some experimentation -may be required. +fork](https://github.com/cloud-hypervisor/edk2/releases/latest). +The edk2 fork carries customizations required to boot AArch64 guests +on cloud-hypervisor. See [docs/uefi.md](docs/uefi.md) for differences +with upstream tianocore/edk2. ### Firmware Booting diff --git a/docs/uefi.md b/docs/uefi.md index cabbdf256..80e635642 100644 --- a/docs/uefi.md +++ b/docs/uefi.md @@ -1,6 +1,40 @@ # UEFI Boot -Cloud Hypervisor supports UEFI boot through the utilization of the EDK II based UEFI firmware. +Cloud Hypervisor supports UEFI boot through the utilization of the EDK II based UEFI firmware. + +## Using Prebuilt UEFI Firmware + +Cloud Hypervisor's [edk2 fork](https://github.com/cloud-hypervisor/edk2) +publishes prebuilt UEFI firmware binaries as release assets. The x86-64 +binary is named `CLOUDHV.fd` and the AArch64 binary is named +`CLOUDHV_EFI.fd`. + +The latest release is always available at +. + +```shell +# x86-64 +$ wget https://github.com/cloud-hypervisor/edk2/releases/latest/download/CLOUDHV.fd + +# AArch64 +$ wget https://github.com/cloud-hypervisor/edk2/releases/latest/download/CLOUDHV_EFI.fd +``` + +Pass the firmware file to `--firmware`. + +```shell +# x86-64 +$ ./cloud-hypervisor --firmware ./CLOUDHV.fd --disk path=guest.raw ... + +# AArch64 +$ ./cloud-hypervisor --firmware ./CLOUDHV_EFI.fd --disk path=guest.raw ... +``` + +Cloud Hypervisor opens the firmware file in read-only mode. + +The sections below describe how to build the firmware from source, which is +only necessary if you need to test edk2 changes or build for a configuration +that the release assets don't cover. ## Building UEFI Firmware for x86-64 @@ -31,67 +65,92 @@ After the successful build, the resulting firmware binaries are available under ## Building UEFI Firmware for AArch64 +Build from Cloud Hypervisor's +[edk2 fork](https://github.com/cloud-hypervisor/edk2), which carries +the submodule revisions and patches needed to boot AArch64 guests on +Cloud Hypervisor. This is the same build that produces the prebuilt +firmware in the fork's release workflow. + ```shell -# On an AArch64 machine: +# On an AArch64 machine. $ sudo apt-get update -$ sudo apt-get install uuid-dev nasm iasl build-essential python3-distutils git -# Master branches for these repos can be unstable, and newer GCC versions -# enforce strict warning-as-error policies that break builds -# These specific commit # are verified to compile cleanly with GCC 13.3.0 -# Shallow clone edk2 repo -$ mkdir -p edk2 && cd edk2 && \ -git init -q && \ -git remote add origin https://github.com/tianocore/edk2.git && \ -git fetch -q --depth 1 origin 22130dcd98b4d4b76ac8d922adb4a2dbc86fa52c && \ -git checkout -q FETCH_HEAD && \ -git submodule update --init --recursive --depth 1 && \ -cd .. -# Shallow clone edk2-platforms repo -$ mkdir -p edk2-platforms && cd edk2-platforms && \ -git init -q && \ -git remote add origin https://github.com/tianocore/edk2-platforms.git && \ -git fetch -q --depth 1 origin 8227e9e9f6a8aefbd772b40138f835121ccb2307 && \ -git checkout -q FETCH_HEAD && \ -cd .. -# Shallow clone acpica repo -$ mkdir -p acpica && cd acpica && \ -git init -q && \ -git remote add origin https://github.com/acpica/acpica.git && \ -git fetch -q --depth 1 origin e80cbd7b52de20aa8c75bfba9845e9cb61f2e681 && \ -git checkout -q FETCH_HEAD && \ -cd .. -# Build tools -$ export PACKAGES_PATH="$PWD/edk2:$PWD/edk2-platforms" -$ export IASL_PREFIX="$PWD/acpica/generate/unix/bin/" -$ make -C acpica -$ cd edk2/ -$ . edksetup.sh -$ cd .. -$ make -C edk2/BaseTools +$ sudo apt-get install uuid-dev iasl build-essential git libbrotli-dev -# Build EDK2 -$ build -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtCloudHv.dsc -b RELEASE +$ git clone --branch ch https://github.com/cloud-hypervisor/edk2.git +$ cd edk2 +$ git submodule update --init --recursive -# Alternate method -# Launch developer container from AArch64 machine -$ ./scripts/dev_cli.sh shell -# Inside the container -$ source scripts/test-util.sh -$ source scripts/common-aarch64.sh -$ build_edk2 +$ source edksetup.sh +$ make -C BaseTools + +$ build -p ArmVirtPkg/ArmVirtCloudHv.dsc -a AARCH64 -t GCC -b RELEASE \ + --pcd gEfiMdeModulePkgTokenSpaceGuid.PcdDxeNxMemoryProtectionPolicy=0xC000000000007FD1 ``` -If the build goes well, the EDK2 binary is available at -`edk2/Build/ArmVirtCloudHv-AARCH64/RELEASE_GCC5/FV/CLOUDHV_EFI.fd` or `workloads/CLOUDHV_EFI.fd` -when using developer container to produce firmware. +The `--pcd` argument keeps `EfiLoaderData` executable so older GRUB +versions can boot, as described in the EfiLoaderData Executability +section below. + +The built firmware is produced at +`Build/ArmVirtCloudHv-AARCH64/RELEASE_GCC/FV/CLOUDHV_EFI.fd`. + +## AArch64 Firmware Notes + +### Multiple PCI Segments + +The AArch64 UEFI firmware (`CLOUDHV_EFI.fd`) uses `FdtPciHostBridgeLib` +to discover PCI host bridges from the device tree. This library only +enumerates the first PCI host bridge (segment 0), regardless of how +many `pci-host-ecam-generic` FDT nodes Cloud Hypervisor provides. +When booting with `--platform num_pci_segments=N` (N > 1), segments +1 through N-1 are not visible to the firmware itself. + +This is not a functional limitation because Cloud Hypervisor provides +ACPI tables (MCFG, DSDT) describing all segments directly to the guest +via `CloudHvAcpiPlatformDxe`. The Linux kernel re-enumerates PCI from +the MCFG table and assigns BARs independently of UEFI. Boot devices +(virtio-blk, virtio-net) must reside on segment 0, which UEFI does +enumerate. + +The DEBUG build configuration additionally asserts and terminates +when more than one `pci-host-ecam-generic` node is present. +The prebuilt `CLOUDHV_EFI.fd` is built with RELEASE configuration to +avoid this assert. If you build the firmware from source and intend +to use multiple PCI segments, build with `-b RELEASE`. + +### EfiLoaderData Executability + +The AArch64 firmware in cloud-hypervisor/edk2 allows code execution +from `EfiLoaderData` memory regions. The upstream tianocore/edk2 +default does the opposite and marks `EfiLoaderData` as non-executable +through the `PcdDxeNxMemoryProtectionPolicy` PCD set in +`ArmVirt.dsc.inc`. The non-executable default was introduced in +upstream commit +[2997ae3873](https://github.com/tianocore/edk2/commit/2997ae3873) +in 2022 and sets the PCD to `0xC000000000007FD5`. + +Keeping `EfiLoaderData` executable is required for older GRUB versions +that allocate their modules into `EfiLoaderData` memory and then +execute code from those allocations. GRUB upstream switched to +`EfiLoaderCode` for this in 2017, but some distributions still ship +the older behavior (e.g. Ubuntu 22.04 / Jammy AArch64 cloud images). +Without the override, those guests fail to boot with an instruction +abort (permission fault, second level). + +The prebuilt `CLOUDHV_EFI.fd` overrides the PCD back to +`0xC000000000007FD1` at build time (clears the `EfiLoaderData` NX +bit). If you build the firmware from source and need to boot guests +with the older GRUB behavior, apply the same override. x86-64 +(`CLOUDHV.fd`) is unaffected as it does not enforce this NX policy. ## Using OVMF Binaries Any UEFI capable image can be booted using the Cloud Hypervisor specific firmware. Windows guests under Cloud Hypervisor only support UEFI boot, therefore OVMF is mandatory there. -To make Cloud Hypervisor use UEFI boot, pass the `CLOUDHV.fd` (for x86-64) / `CLOUDHV_EFI.fd` (for AArch64) file path as an argument to the `--kernel` option. The firmware file will be opened in read only mode. +To make Cloud Hypervisor use UEFI boot, pass the `CLOUDHV.fd` (for x86-64) or `CLOUDHV_EFI.fd` (for AArch64) file path as an argument to the `--firmware` option, which opens the firmware file in read-only mode. # Links -- [OVMF wiki](https://github.com/tianocore/tianocore.github.io/wiki/OVMF) -- [Cloud Hypervisor specific tree](https://github.com/cloud-hypervisor/edk2/tree/ch) +- [OVMF wiki](https://github.com/tianocore/tianocore.github.io/wiki/OVMF) +- [Cloud Hypervisor edk2 fork](https://github.com/cloud-hypervisor/edk2) +- [Cloud Hypervisor edk2 releases](https://github.com/cloud-hypervisor/edk2/releases)