containerd/docs/user-namespaces/README.md

Support for user namespaces

Kubernetes supports running pods with user namespace since v1.25. This document explains the containerd support for this feature.

What are user namespaces?

A user namespace isolates the user running inside the container from the one in the host.

A process running as root in a container can run as a different (non-root) user in the host; in other words, the process has full privileges for operations inside the user namespace, but is unprivileged for operations outside the namespace.

You can use this feature to reduce the damage a compromised container can do to the host or other pods in the same node. There are several security vulnerabilities rated either HIGH or CRITICAL that were not exploitable when user namespaces is active. It is expected user namespace will mitigate some future vulnerabilities too.

See the kubernetes documentation for a high-level introduction to user namespaces.

Stack requirements

The Kubernetes implementation was redesigned in 1.27, so the requirements are different for versions pre and post Kubernetes 1.27.

Please note that if you try to use user namespaces with containerd 1.6 or older, the hostUsers: false setting in your pod.spec will be silently ignored.

Kubernetes 1.25 and 1.26

• Containerd 1.7
• You can use runc or crun as the OCI runtime:
  • runc 1.1 or greater
  • crun 1.4.3 or greater

You can also use containerd 2.0 or above, but the same requirements as Kubernetes 1.27 and greater apply, except for the Linux kernel. Bear in mind that all the requirements there apply, including file-systems supporting idmap mounts. You can use Linux versions:

• Linux 5.15: you will suffer from the containerd 1.7 storage and latency limitations, as it doesn't support idmap mounts for overlayfs.
• Linux 5.19 or greater (recommended): it doesn't suffer from any of the containerd 1.7 limitations, as overlayfs started supporting idmap mounts on this kernel version.

Kubernetes 1.27 and greater

• Linux 6.3 or greater
• Containerd 2.0 or greater
• You can use runc or crun as the OCI runtime:
  • runc 1.2 or greater
  • crun 1.9 or greater

Furthermore, all the file-systems used by the volumes in the pod need kernel-support for idmap mounts. Some popular file-systems that support idmap mounts in Linux 6.3 are: btrfs, ext4, xfs, fat, tmpfs, overlayfs.

The kubelet is in charge of populating some files to the containers (like configmap, secrets, etc.). The file-system used in that path needs to support idmap mounts too. See the Kubernetes documentation for more info on that.

Creating a Kubernetes pod with user namespaces

First check your containerd, Linux and Kubernetes versions. If those are okay, then there is no special configuration needed on conntainerd. You can just follow the steps in the Kubernetes website.

Limitations

You can check the limitations Kubernetes has here. Note that different Kubernetes versions have different limitations, be sure to check the site for the Kubernetes version you are using.

Different containerd versions have different limitations too, those are highlighted in this section.

containerd 1.7

One limitation present in containerd 1.7 is that it needs to change the ownership of every file and directory inside the container image, during Pod startup. This means it has a storage overhead (the size of the container image is duplicated each time a pod is created) and can significantly impact the container startup latency.

You can mitigate this limitation by switching /sys/module/overlay/parameters/metacopy to Y. This will significantly reduce the storage and performance overhead, as only the inode for each file of the container image will be duplicated, but not the content of the file. This means it will use less storage and it will be faster. However, it is not a panacea.

If you change the metacopy param, make sure to do it in a way that is persistant across reboots. You should also be aware that this setting will be used for all containers, not just containers with user namespaces enabled. This will affect all the snapshots that you take manually (if you happen to do that). In that case, make sure to use the same value of /sys/module/overlay/parameters/metacopy when creating and restoring the snapshot.

containerd 2.0

The storage and latency limitation from containerd 1.7 are not present in container 2.0 and above, if you use the overlay snapshotter (this is used by default). It will not use more storage at all, and there is no startup latency.

This is achieved by using the kernel feature idmap mounts with the container rootfs (the container image). This allows an overlay file-system to expose the image with different UID/GID without copying the files nor the inodes, just using a bind-mount.

You can check if you are using idmap mounts for the container image if you create a pod with user namespaces, exec into it and run:

mount | grep overlay

You should see a reference to the idmap mount in the lowerdir parameter, in this case we can see idmapped used there:

overlay on / type overlay (rw,relatime,lowerdir=/tmp/ovl-idmapped823885363/0,upperdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1018/fs,workdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1018/work)

Creating a container with user namespaces with ctr

You can also create a container with user namespaces using ctr. This is more low-level, be warned.

Create an OCI bundle as explained here. Then, change the UID/GID to 65536:

sudo chown -R 65536:65536 rootfs/

Copy this config.json and replace XXX-path-to-rootfs with the absolute path to the rootfs you just chowned.

Then create and start the container with:

sudo ctr create --config <path>/config.json userns-test
sudo ctr t start userns-test

This will open a shell inside the container. You can run this, to verify you are inside a user namespace:

root@runc:/# cat /proc/self/uid_map
         0      65536      65536

The output should be exactly the same.