cloud-hypervisor/docs/live_migration.md

# Live Migration

This document gives examples of how to use the live migration support
in Cloud Hypervisor:

1. **Local Migration**: Migrating a VM from one Cloud Hypervisor instance to another on the same machine; also called
  UNIX socket migration.
1. **Remote Migration** (TCP Migration): migrating a VM between two TCP/IP hosts.

> :warning: These examples place sockets in /tmp. This is done for
> simplicity and should not be done in production.

## Local Migration (Suitable for Live Upgrade of VMM)

Launch the source VM (on the host machine):

```console
$ target/release/cloud-hypervisor
    --kernel ~/workloads/vmlinux \
    --disk path=~/workloads/focal.raw \
    --cpus boot=1 --memory size=1G,shared=on \
    --cmdline "root=/dev/vda1 console=ttyS0"  \
    --serial tty --console off --api-socket=/tmp/api1
```

Launch the destination VM from the same directory (on the host machine):

```console
$ target/release/cloud-hypervisor --api-socket=/tmp/api2
```

Get ready for receiving migration for the destination VM (on the host
machine):

```console
$ target/release/ch-remote --api-socket=/tmp/api2 receive-migration receiver_url=unix:/tmp/sock
```

Start to send migration for the source VM (on the host machine):

```console
$ target/release/ch-remote --api-socket=/tmp/api1 send-migration destination_url=unix:/tmp/sock,local=on
```

When the above commands completed, the source VM should be successfully
migrated to the destination VM. Now the destination VM is running while
the source VM is terminated gracefully.

## Remote Migration (TCP Migration)

_Hint: For developing purposes, same-host TCP migrations are also supported._

In this example, we will migrate a VM from one machine (`src`) to
another (`dst`) across the network. To keep it simple, we will use a
minimal VM setup without storage.

### Preparation

Make sure that `src` and `dst` can reach each other via the
network. You should be able to ping each machine. Also each machine
should have an open TCP port.

You will need a kernel and initramfs for a minimal Linux system. For
this example, we will use the Debian netboot image.

Place the kernel and initramfs into the _same directory_ on both
machines. This is important for the migration to succeed. We will use
`/var/images`:

```console
src $ export DEBIAN=https://ftp.debian.org/debian/dists/stable/main/installer-amd64/current/images/netboot/debian-installer/amd64
src $ mkdir -p /var/images
src $ curl $DEBIAN/linux > /var/images/linux
src $ curl $DEBIAN/initrd.gz > /var/images/initrd
```

Repeat the above steps on the destination host.

### Unix Socket Migration

If Unix socket is selected for migration, we can tunnel traffic through "socat".

#### Starting the Receiver VM

On the receiver side, we prepare an empty VM:

```console
dst $ cloud-hypervisor --api-socket /tmp/api
```

In a different terminal, configure the VM as a migration target:

```console
dst $ ch-remote --api-socket=/tmp/api receive-migration receiver_url=unix:/tmp/sock
```

In yet another terminal, forward TCP connections to the Unix domain socket:

```console
dst $ socat TCP-LISTEN:{port},reuseaddr UNIX-CLIENT:/tmp/sock
```

#### Starting the Sender VM

Let's start the VM on the source machine:

```console
src $ cloud-hypervisor \
        --serial tty --console off \
        --cpus boot=2 --memory size=4G \
        --kernel /var/images/linux \
        --initramfs /var/images/initrd \
        --cmdline "console=ttyS0" \
        --api-socket /tmp/api
```

After a few seconds the VM should be up and you can interact with it.

#### Performing the Migration

First, we start `socat`:

```console
src $ socat UNIX-LISTEN:/tmp/sock,reuseaddr TCP:{dst}:{port}
```

> Replace {dst}:{port} with the actual IP address and port of your destination host.

Then we kick-off the migration itself:

```console
src $ ch-remote --api-socket=/tmp/api send-migration destination_url=unix:/tmp/sock
```

When the above commands completed, the VM should be successfully
migrated to the destination machine without interrupting the workload.

### Network Announcements After Resume

After a VM resumes from migration, snapshot restore, or any other path
that restores a previously paused VM, Cloud Hypervisor asks supported
network devices to announce the VM from its new host. For `virtio-net`,
the current implementation sets `VIRTIO_NET_S_ANNOUNCE`, raises a config
interrupt, retries that request a few times in the background, and also
sends host-side RARP announcements on the TAP interfaces. A guest
re-announcement therefore only happens when the guest negotiated
`VIRTIO_NET_F_GUEST_ANNOUNCE`. For `vhost-user-net`, the current implementation
only uses the guest announcement path.

### TCP Socket Migration

If TCP socket is selected for migration, we need to consider migrating
in a trusted network.

#### Starting the Receiver VM

On the receiver side, we prepare an empty VM:

```console
dst $ cloud-hypervisor --api-socket /tmp/api
```

In a different terminal, prepare to receive the migration:

```console
dst $ ch-remote --api-socket=/tmp/api receive-migration receiver_url=tcp:0.0.0.0:{port}
```

#### Starting the Sender VM

Let's start the VM on the source machine:

```console
src $ cloud-hypervisor \
        --serial tty --console off \
        --cpus boot=2 --memory size=4G \
        --kernel /var/images/linux \
        --initramfs /var/images/initrd \
        --cmdline "console=ttyS0" \
        --api-socket /tmp/api
```

After a few seconds the VM should be up and you can interact with it.

#### Performing the Migration

Initiate the Migration over TCP:

```console
src $ ch-remote --api-socket=/tmp/api send-migration destination_url=tcp:{dst}:{port}
```

With migration parameters:

```console
src $ ch-remote --api-socket=/tmp/api send-migration destination_url=tcp:{dst}:{port},downtime_ms=200,timeout_s=3600,timeout_strategy=cancel
```

> Replace {dst}:{port} with the actual IP address and port of your destination host.

After completing the above commands, the source VM will be migrated to
the destination host and continue running there. The source VM instance
will terminate normally. All ongoing processes and connections within
the VM should remain intact after the migration.
See [Network Announcements After Resume](#network-announcements-after-resume)
for the announcement behavior after a VM resumes.

#### Encryption

TCP migration can be protected with TLS by passing `tls_dir=<path>` to
both `receive-migration` and `send-migration`.

The destination host needs a directory containing:

- `ca-cert.pem`: the CA certificate used to verify source certificates
- `server-cert.pem`: the certificate presented by the destination
- `server-key.pem`: the private key for `server-cert.pem`

The source host needs a directory containing:

- `ca-cert.pem`: the CA certificate used to verify the destination
  certificate
- `client-cert.pem`: the certificate presented by the source
- `client-key.pem`: the private key for `client-cert.pem`

Protect the private key files with file mode `0600` to reduce the risk
of accidental disclosure:

```console
$ chmod 600 server-key.pem client-key.pem
```

Current TCP migration uses mutual TLS (mTLS) authentication. The source
verifies the destination certificate against `ca-cert.pem` and presents
`client-cert.pem` and `client-key.pem`. The destination presents
`server-cert.pem` and `server-key.pem`, and only accepts client
certificates that chain to `ca-cert.pem`.

Example receiver command:

```console
dst $ ch-remote --api-socket=/tmp/api receive-migration receiver_url=tcp:0.0.0.0:{port},tls_dir=/path/to/dst-tls
```

Example sender command:

```console
src $ ch-remote --api-socket=/tmp/api send-migration destination_url=tcp:{dst}:{port},tls_dir=/path/to/src-tls
```

TLS encryption is only supported with `tcp:<host>:<port>` migration
URLs, not with local UNIX-socket migration.

The commands below describe how to create the encryption material necessary
to perform a same-host TCP migration with TLS.

##### Setup the Certificate Authority

First, set up the CA with a private key and a self-signed certificate.

```console
$ certtool --generate-privkey \
           --outfile ca-key.pem
```

To generate the certificate, provide at least your organization name in a template file, for example `ca.info`:

```text
cn = Name of your organization
ca
cert_signing_key
```

Then you can create the certificate:

```console
$ certtool --generate-self-signed \
           --load-privkey ca-key.pem \
           --template ca.info \
           --outfile ca-cert.pem
```

##### Issuing host certificates

Generate a private key and a certificate for each side. Cloud Hypervisor enforces mTLS, thus you also need a key for the migration sender.

The certificates also require template data. You can use separate templates for sender and receiver, but this example uses a single template, `hosts.info`:

```text
organization = Name of your organization
cn = localhost
tls_www_server
tls_www_client
signing_key
dns_name = localhost
ip_address = 127.0.0.1
```

Then you can create the necessary files for the client (migration sender):

```console
$ certtool --generate-privkey \
           --outfile client-key.pem

$ certtool --generate-certificate \
           --load-ca-certificate ca-cert.pem \
           --load-ca-privkey ca-key.pem \
           --load-privkey client-key.pem \
           --template hosts.info \
           --outfile client-cert.pem
```

And the necessary files for the server (migration receiver):

```console
$ certtool --generate-privkey \
           --outfile server-key.pem

$ certtool --generate-certificate \
           --load-ca-certificate ca-cert.pem \
           --load-ca-privkey ca-key.pem \
           --load-privkey server-key.pem \
           --template hosts.info \
           --outfile server-cert.pem
```

For a same-host test, place these files in a single directory and use that directory as `tls_dir` for both commands.

##### Performing the Migration

On the receiver side, we prepare an empty VM:

```console
dst $ cloud-hypervisor --api-socket /tmp/api-rcv
```

In a different terminal, prepare to receive the migration:

```console
dst $ ch-remote --api-socket=/tmp/api-rcv receive-migration receiver_url=tcp:0.0.0.0:9000,tls_dir=/path/to/certificates
```

On the sender side, we start a VM:

```console
src $ cloud-hypervisor \
        --serial tty --console off \
        --cpus boot=2 --memory size=4G \
        --kernel /var/images/linux \
        --initramfs /var/images/initrd \
        --cmdline "console=ttyS0" \
        --api-socket /tmp/api-src
```

After a few seconds the VM should be up and you can interact with it. Then trigger the migration:

```console
src $ ch-remote --api-socket=/tmp/api-src send-migration destination_url=tcp:localhost:9000,tls_dir=/path/to/certificates
```

#### Migration Parameters

Cloud Hypervisor supports additional parameters to control the
migration process. Via the API or `ch-remote`, you may specify:

- `downtime_ms <milliseconds>`: \
  The maximum downtime the migration aims for, in milliseconds.
  Defaults to `300ms`.
- `timeout_s <seconds>`: \
  The timeout for the migration (maximum total duration), in seconds.
  Defaults to `3600s` (one hour).
- `timeout_strategy <strategy>` (`[cancel, ignore]`): \
  The strategy to apply when the migration timeout is reached.
  Cancel will abort the migration and keep the VM running on the source.
  Ignore will proceed with the migration regardless of the downtime requirement.
  Defaults to `cancel`.
- `connections <amount>`: \
  The number of parallel TCP connections to use for migration.
  Must be between `1` and `128`. Defaults to `1`.
  Multiple connections are not supported with local UNIX-socket migration.
- `memory_mode <precopy|postcopy>`: \
  Memory transfer mode. `postcopy` resumes the destination first and faults
  guest pages in on demand over a dedicated connection. Defaults to `precopy`.

## Migrating VMs with VFIO Devices

VMs with migratable VFIO devices, for example NIC VFs bound to
`mlx5_vfio_pci`, can be live migrated. The opaque device state is captured
during the downtime window and the destination receives new device file
descriptors through the `vfio_fds` and `iommufd_fd` options of
`receive-migration`.

See [Live Migration of VFIO Devices](vfio.md#live-migration-of-vfio-devices)
for the requirements, the dirty tracking behavior, and an example.

## Version Compatibility

Cloud Hypervisor live migration compatibility has two dimensions: the
Cloud Hypervisor version and the migration protocol version.

Cloud Hypervisor uses a versioned migration protocol for the messages exchanged
between source and destination. Each Cloud Hypervisor release sends its current
migration protocol version and accepts that version plus the immediately
previous protocol version.

This means migration is supported from an older protocol version to the same or
next protocol version. If the source and destination are more than one migration
protocol version apart, the VM must be migrated through an intermediate
Cloud Hypervisor version first.

### Technical Details

The source sends its migration protocol version in the initial `Start` request.
The destination accepts the migration only if that protocol version is supported.
A zeroed `Start` command header is handled as protocol `v0`, so older
deployments that do not explicitly send a protocol version remain compatible.

The migration protocol version covers the protocol spoken after a migration
connection has been established. Examples include adding a mandatory migration
command, changing the order of migration protocol messages, or changing the
framing or encoding of protocol command payloads.

The migration protocol version does not cover migration transport setup. For
example, choosing TCP vs. UNIX sockets or opening the initial connection needs
separate compatibility handling.

Migration protocol versioning is separate from snapshot state compatibility.
Device and VM state changes still need to be handled by the respective snapshot
serialization/deserialization code. That compatibility is Cloud Hypervisor
version dependent, but it is not the responsibility of migration protocol
versioning.