doc: Add documentation about CRI user namespaces
Signed-off-by: Rodrigo Campos <rodrigoca@microsoft.com>
This commit is contained in:
Rodrigo Campos
parent
2e13d39546
commit
967313049f
@ -461,4 +461,4 @@ more quickly.
|
|||||||
| [NRI in CRI Support](https://github.com/containerd/containerd/pull/6019) | containerd v1.7 | containerd v2.0 |
|
| [NRI in CRI Support](https://github.com/containerd/containerd/pull/6019) | containerd v1.7 | containerd v2.0 |
|
||||||
| [gRPC Shim](https://github.com/containerd/containerd/pull/8052) | containerd v1.7 | containerd v2.0 |
|
| [gRPC Shim](https://github.com/containerd/containerd/pull/8052) | containerd v1.7 | containerd v2.0 |
|
||||||
| [CRI Runtime Specific Snapshotter](https://github.com/containerd/containerd/pull/6899) | containerd v1.7 | containerd v2.0 |
|
| [CRI Runtime Specific Snapshotter](https://github.com/containerd/containerd/pull/6899) | containerd v1.7 | containerd v2.0 |
|
||||||
| [CRI Support for User Namespaces](https://github.com/containerd/containerd/pull/7679) | containerd v1.7 | containerd v2.0 |
|
| [CRI Support for User Namespaces](./docs/user-namespaces/README.md) | containerd v1.7 | containerd v2.0 |
|
||||||
|
|||||||
146
docs/user-namespaces/README.md
Normal file
146
docs/user-namespaces/README.md
Normal file
@ -0,0 +1,146 @@
|
|||||||
|
# Support for user namespaces
|
||||||
|
|
||||||
|
Kubernetes supports running pods with user namespace since v1.25. This document explains the
|
||||||
|
containerd support for this feature.
|
||||||
|
|
||||||
|
## What are user namespaces?
|
||||||
|
|
||||||
|
A user namespace isolates the user running inside the container from the one in the host.
|
||||||
|
|
||||||
|
A process running as root in a container can run as a different (non-root) user in the host; in
|
||||||
|
other words, the process has full privileges for operations inside the user namespace, but is
|
||||||
|
unprivileged for operations outside the namespace.
|
||||||
|
|
||||||
|
You can use this feature to reduce the damage a compromised container can do to the host or other
|
||||||
|
pods in the same node. There are several security vulnerabilities rated either HIGH or CRITICAL that
|
||||||
|
were not exploitable when user namespaces is active. It is expected user namespace will mitigate
|
||||||
|
some future vulnerabilities too.
|
||||||
|
|
||||||
|
See [the kubernetes documentation][kube-intro] for a high-level introduction to
|
||||||
|
user namespaces.
|
||||||
|
|
||||||
|
[kube-intro]: https://kubernetes.io/docs/concepts/workloads/pods/user-namespaces/#introduction
|
||||||
|
|
||||||
|
## Stack requirements
|
||||||
|
|
||||||
|
The Kubernetes implementation was redesigned in 1.27, so the requirements are different for versions
|
||||||
|
pre and post Kubernetes 1.27.
|
||||||
|
|
||||||
|
Please note that if you try to use user namespaces with containerd 1.6 or older, the `hostUsers:
|
||||||
|
false` setting in your pod.spec will be **silently ignored**.
|
||||||
|
|
||||||
|
### Kubernetes 1.25 and 1.26
|
||||||
|
|
||||||
|
* Containerd 1.7 or greater
|
||||||
|
* runc 1.1 or greater
|
||||||
|
|
||||||
|
### Kubernetes 1.27 and greater
|
||||||
|
|
||||||
|
* Linux 6.3 or greater
|
||||||
|
* Containerd 2.0 or greater
|
||||||
|
* You can use runc or crun as the OCI runtime:
|
||||||
|
* runc 1.2 or greater
|
||||||
|
* crun 1.9 or greater
|
||||||
|
|
||||||
|
Furthermore, all the file-systems used by the volumes in the pod need kernel-support for idmap
|
||||||
|
mounts. Some popular file-systems that support idmap mounts in Linux 6.3 are: `btrfs`, `ext4`, `xfs`,
|
||||||
|
`fat`, `tmpfs`, `overlayfs`.
|
||||||
|
|
||||||
|
The kubelet is in charge of populating some files to the containers (like configmap, secrets, etc.).
|
||||||
|
The file-system used in that path needs to support idmap mounts too. See [the Kubernetes
|
||||||
|
documentation][kube-req] for more info on that.
|
||||||
|
|
||||||
|
|
||||||
|
[kube-req]: https://kubernetes.io/docs/concepts/workloads/pods/user-namespaces/#before-you-begin
|
||||||
|
|
||||||
|
## Creating a Kubernetes pod with user namespaces
|
||||||
|
|
||||||
|
First check your containerd, Linux and Kubernetes versions. If those are okay, then there is no
|
||||||
|
special configuration needed on conntainerd. You can just follow the steps in the [Kubernetes
|
||||||
|
website][kube-example].
|
||||||
|
|
||||||
|
[kube-example]: https://kubernetes.io/docs/tasks/configure-pod-container/user-namespaces/
|
||||||
|
|
||||||
|
# Limitations
|
||||||
|
|
||||||
|
You can check the limitations Kubernetes has [here][kube-limitations]. Note that different
|
||||||
|
Kubernetes versions have different limitations, be sure to check the site for the Kubernetes version
|
||||||
|
you are using.
|
||||||
|
|
||||||
|
Different containerd versions have different limitations too, those are highlighted in this section.
|
||||||
|
|
||||||
|
[kube-limitations]: https://kubernetes.io/docs/concepts/workloads/pods/user-namespaces/#limitations
|
||||||
|
|
||||||
|
### containerd 1.7
|
||||||
|
|
||||||
|
One limitation present in containerd 1.7 is that it needs to change the ownership of every file and
|
||||||
|
directory inside the container image, during Pod startup. This means it has a storage overhead (the
|
||||||
|
size of the container image is duplicated each time a pod is created) and can significantly impact
|
||||||
|
the container startup latency.
|
||||||
|
|
||||||
|
You can mitigate this limitation by switching `/sys/module/overlay/parameters/metacopy` to `Y`. This
|
||||||
|
will significantly reduce the storage and performance overhead, as only the inode for each file of
|
||||||
|
the container image will be duplicated, but not the content of the file. This means it will use less
|
||||||
|
storage and it will be faster. However, it is not a panacea.
|
||||||
|
|
||||||
|
If you change the metacopy param, make sure to do it in a way that is persistant across reboots. You
|
||||||
|
should also be aware that this setting will be used for all containers, not just containers with
|
||||||
|
user namespaces enabled. This will affect all the snapshots that you take manually (if you happen to
|
||||||
|
do that). In that case, make sure to use the same value of `/sys/module/overlay/parameters/metacopy`
|
||||||
|
when creating and restoring the snapshot.
|
||||||
|
|
||||||
|
### containerd 2.0
|
||||||
|
|
||||||
|
The storage and latency limitation from containerd 1.7 are not present in container 2.0 and above,
|
||||||
|
if you use the overlay snapshotter (this is used by default). It will not use more storage at all,
|
||||||
|
and there is no startup latency.
|
||||||
|
|
||||||
|
This is achieved by using the kernel feature idmap mounts with the container rootfs (the container
|
||||||
|
image). This allows an overlay file-system to expose the image with different UID/GID without copying
|
||||||
|
the files nor the inodes, just using a bind-mount.
|
||||||
|
|
||||||
|
You can check if you are using idmap mounts for the container image if you create a pod with user
|
||||||
|
namespaces, exec into it and run:
|
||||||
|
|
||||||
|
```
|
||||||
|
mount | grep overlay
|
||||||
|
```
|
||||||
|
|
||||||
|
You should see a reference to the idmap mount in the `lowerdir` parameter, in this case we can see
|
||||||
|
`idmapped` used there:
|
||||||
|
|
||||||
|
```
|
||||||
|
overlay on / type overlay (rw,relatime,lowerdir=/tmp/ovl-idmapped823885363/0,upperdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1018/fs,workdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1018/work)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Creating a container with user namespaces with `ctr`
|
||||||
|
|
||||||
|
You can also create a container with user namespaces using `ctr`. This is more low-level, be warned.
|
||||||
|
|
||||||
|
Create an OCI bundle as explained [here][runc-bundle]. Then, change the UID/GID to 65536:
|
||||||
|
|
||||||
|
```
|
||||||
|
sudo chown -R 65536:65536 rootfs/
|
||||||
|
```
|
||||||
|
|
||||||
|
Copy [this config.json](./config.json) and replace `XXX-path-to-rootfs` with the
|
||||||
|
absolute path to the rootfs you just chowned.
|
||||||
|
|
||||||
|
Then create and start the container with:
|
||||||
|
|
||||||
|
```
|
||||||
|
sudo ctr create --config <path>/config.json userns-test
|
||||||
|
sudo ctr t start userns-test
|
||||||
|
```
|
||||||
|
|
||||||
|
This will open a shell inside the container. You can run this, to verify you are inside a user
|
||||||
|
namespace:
|
||||||
|
|
||||||
|
```
|
||||||
|
root@runc:/# cat /proc/self/uid_map
|
||||||
|
0 65536 65536
|
||||||
|
```
|
||||||
|
|
||||||
|
The output should be exactly the same.
|
||||||
|
|
||||||
|
[runc-bundle]: https://github.com/opencontainers/runc#creating-an-oci-bundle
|
||||||
199
docs/user-namespaces/config.json
Normal file
199
docs/user-namespaces/config.json
Normal file
@ -0,0 +1,199 @@
|
|||||||
|
{
|
||||||
|
"ociVersion": "1.0.2-dev",
|
||||||
|
"process": {
|
||||||
|
"terminal": true,
|
||||||
|
"user": {
|
||||||
|
"uid": 0,
|
||||||
|
"gid": 0
|
||||||
|
},
|
||||||
|
"args": [
|
||||||
|
"bash"
|
||||||
|
],
|
||||||
|
"env": [
|
||||||
|
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
|
||||||
|
"TERM=xterm"
|
||||||
|
],
|
||||||
|
"cwd": "/",
|
||||||
|
"capabilities": {
|
||||||
|
"bounding": [
|
||||||
|
"CAP_AUDIT_WRITE",
|
||||||
|
"CAP_KILL",
|
||||||
|
"CAP_NET_BIND_SERVICE"
|
||||||
|
],
|
||||||
|
"effective": [
|
||||||
|
"CAP_AUDIT_WRITE",
|
||||||
|
"CAP_KILL",
|
||||||
|
"CAP_NET_BIND_SERVICE"
|
||||||
|
],
|
||||||
|
"inheritable": [
|
||||||
|
"CAP_AUDIT_WRITE",
|
||||||
|
"CAP_KILL",
|
||||||
|
"CAP_NET_BIND_SERVICE"
|
||||||
|
],
|
||||||
|
"permitted": [
|
||||||
|
"CAP_AUDIT_WRITE",
|
||||||
|
"CAP_KILL",
|
||||||
|
"CAP_NET_BIND_SERVICE"
|
||||||
|
],
|
||||||
|
"ambient": [
|
||||||
|
"CAP_AUDIT_WRITE",
|
||||||
|
"CAP_KILL",
|
||||||
|
"CAP_NET_BIND_SERVICE"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"rlimits": [
|
||||||
|
{
|
||||||
|
"type": "RLIMIT_NOFILE",
|
||||||
|
"hard": 1024,
|
||||||
|
"soft": 1024
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"noNewPrivileges": true
|
||||||
|
},
|
||||||
|
"root": {
|
||||||
|
"path": "XXX-path-to-rootfs"
|
||||||
|
},
|
||||||
|
"hostname": "runc",
|
||||||
|
"mounts": [
|
||||||
|
{
|
||||||
|
"destination": "/proc",
|
||||||
|
"type": "proc",
|
||||||
|
"source": "proc"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"destination": "/dev",
|
||||||
|
"type": "tmpfs",
|
||||||
|
"source": "tmpfs",
|
||||||
|
"options": [
|
||||||
|
"nosuid",
|
||||||
|
"strictatime",
|
||||||
|
"mode=755",
|
||||||
|
"size=65536k"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"destination": "/dev/pts",
|
||||||
|
"type": "devpts",
|
||||||
|
"source": "devpts",
|
||||||
|
"options": [
|
||||||
|
"nosuid",
|
||||||
|
"noexec",
|
||||||
|
"newinstance",
|
||||||
|
"ptmxmode=0666",
|
||||||
|
"mode=0620",
|
||||||
|
"gid=5"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"destination": "/dev/shm",
|
||||||
|
"type": "tmpfs",
|
||||||
|
"source": "shm",
|
||||||
|
"options": [
|
||||||
|
"nosuid",
|
||||||
|
"noexec",
|
||||||
|
"nodev",
|
||||||
|
"mode=1777",
|
||||||
|
"size=65536k"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"destination": "/dev/mqueue",
|
||||||
|
"type": "mqueue",
|
||||||
|
"source": "mqueue",
|
||||||
|
"options": [
|
||||||
|
"nosuid",
|
||||||
|
"noexec",
|
||||||
|
"nodev"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"destination": "/sys",
|
||||||
|
"type": "sysfs",
|
||||||
|
"source": "sysfs",
|
||||||
|
"options": [
|
||||||
|
"nosuid",
|
||||||
|
"noexec",
|
||||||
|
"nodev",
|
||||||
|
"ro"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"destination": "/sys/fs/cgroup",
|
||||||
|
"type": "cgroup",
|
||||||
|
"source": "cgroup",
|
||||||
|
"options": [
|
||||||
|
"nosuid",
|
||||||
|
"noexec",
|
||||||
|
"nodev",
|
||||||
|
"relatime",
|
||||||
|
"ro"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"linux": {
|
||||||
|
"uidMappings": [
|
||||||
|
{
|
||||||
|
"containerID": 0,
|
||||||
|
"hostID": 65536,
|
||||||
|
"size": 65536
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"gidMappings": [
|
||||||
|
{
|
||||||
|
"containerID": 0,
|
||||||
|
"hostID": 65536,
|
||||||
|
"size": 65536
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"resources": {
|
||||||
|
"devices": [
|
||||||
|
{
|
||||||
|
"allow": false,
|
||||||
|
"access": "rwm"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"namespaces": [
|
||||||
|
{
|
||||||
|
"type": "pid"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "network"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "ipc"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "uts"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "mount"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "cgroup"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "user"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"maskedPaths": [
|
||||||
|
"/proc/acpi",
|
||||||
|
"/proc/asound",
|
||||||
|
"/proc/kcore",
|
||||||
|
"/proc/keys",
|
||||||
|
"/proc/latency_stats",
|
||||||
|
"/proc/timer_list",
|
||||||
|
"/proc/timer_stats",
|
||||||
|
"/proc/sched_debug",
|
||||||
|
"/sys/firmware",
|
||||||
|
"/proc/scsi"
|
||||||
|
],
|
||||||
|
"readonlyPaths": [
|
||||||
|
"/proc/bus",
|
||||||
|
"/proc/fs",
|
||||||
|
"/proc/irq",
|
||||||
|
"/proc/sys",
|
||||||
|
"/proc/sysrq-trigger"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
Loading…
Reference in New Issue
Block a user