3742f7f0db
It's followup for #5890. The containerd-shim process depends on the mount package to init rootfs for container. For the container enable user namespace, the mount package needs to fork child process to get the brand-new user namespace. However, there are two reapers in one process (described by the following list) and there are race-condition cases. 1. mount package 2. sys.Reaper as global one which watch all the SIGCHLD. === [kill(2)][kill] the wrong process === Currently, we use pipe to ensure that child process is alive. However, the pide file descriptor can be hold by other process, which the child process cannot exit by self. We should use [kill(2)][kill] to ensure the child process. But we might kill the wrong process if the child process might be reaped by containerd-shim and the PID might be reused by other process. === [waitid(2)][waitid] on the wrong child process === ``` containerd-shim process: Goroutine 1(GetUsernsFD): Goroutine 2(Reaper) 1. Ready to wait for child process X 2. Received SIGCHLD from X 3. Reaped the zombie child process X (X has been reused by other child process) 4. Wait on process X The goroutine 1 will be stuck until the process X has been terminated. ``` === open `/proc/X/ns/user` on the wrong child process === There is also pid-reused risk between opening `/proc/$pid/ns/user` and writing `/proc/$pid/u[g]id_map`. ``` containerd-shim process: Goroutine 1(GetUsernsFD): Goroutine 2(Reaper) 1. Fork child process X 2. Write /proc/X/uid_map,gid_map 3. Received SIGCHLD from X 4. Reaped the zombie child process X (X has been reused by other process) 5. Open /proc/X/ns/user file as usernsFD The usernsFD links to the wrong X!!! ``` In order to fix the race-condition, we should use [CLONE_PIDFD][clone2] (Since Linux v5.2). When we fork child process `X`, the kernel will return a process file descriptor `X_PIDFD` referencing to child process `X`. With the pidfd, we can use [pidfd_send_signal(2)][pidfd_send_signal] (Since Linux v5.1) to send signal(0) to ensure the child process `X` is alive. If the `X` has terminated and its PID has been recycled for another process. The pidfd_send_signal fails with the error ESRCH. Therefore, we can open `/proc/X/{ns/user,uid_map,gid_map}` file descriptors as first and then use pidfd_send_signal to check the process is still alive. If so, we can ensure the file descriptors are valid and reference to the child process `X`. Even if the `X` PID has been reused after pidfd_send_signal call, the file descriptors are still valid. ```code X, pidfd = clone2(CLONE_PIDFD) usernsFD = open /proc/X/ns/user uidmapFD = open /proc/X/uid_map gidmapFD = open /proc/X/gid_map pidfd_send_signal pidfd, signal(0) return err if no such process == When we arrive here, we can ensure usernsFD/uidmapFD/gidmapFD are correct == even if X has been reused after pidfd_send_signal call. update uid/gid mapping by uidmapFD/gidmapFD return usernsFD ``` And the [waitid(2)][waitid] also supports pidfd type (Since Linux 5.4). We can use pidfd type waitid to ensure we are waiting for the correct process. All the PID related race-condition issues can be resolved by pidfd. ```bash ➜ mount git:(followup-idmapped) pwd /home/fuwei/go/src/github.com/containerd/containerd/mount ➜ mount git:(followup-idmapped) sudo go test -test.root -run TestGetUsernsFD -count=1000 -failfast -p 100 ./... PASS ok github.com/containerd/containerd/mount 3.446s ``` [kill]: <https://man7.org/linux/man-pages/man2/kill.2.html> [clone2]: <https://man7.org/linux/man-pages/man2/clone.2.html> [pidfd_send_signal]: <https://man7.org/linux/man-pages/man2/pidfd_send_signal.2.html> [waitid]: <https://man7.org/linux/man-pages/man2/waitid.2.html> Signed-off-by: Wei Fu <fuweid89@gmail.com>
95 lines
2.5 KiB
Go
95 lines
2.5 KiB
Go
/*
|
|
Copyright The containerd Authors.
|
|
|
|
Licensed under the Apache License, Version 2.0 (the "License");
|
|
you may not use this file except in compliance with the License.
|
|
You may obtain a copy of the License at
|
|
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
Unless required by applicable law or agreed to in writing, software
|
|
distributed under the License is distributed on an "AS IS" BASIS,
|
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
See the License for the specific language governing permissions and
|
|
limitations under the License.
|
|
*/
|
|
|
|
package sys
|
|
|
|
import (
|
|
"runtime"
|
|
"syscall"
|
|
"unsafe"
|
|
)
|
|
|
|
// ForkUserns is to fork child process with user namespace. It returns child
|
|
// process's pid and pidfd reference to the child process.
|
|
//
|
|
// Precondition: The runtime OS thread must be locked, which is GO runtime
|
|
// requirement.
|
|
//
|
|
// Beside this, the child process sets PR_SET_PDEATHSIG with SIGKILL so that
|
|
// the parent process's OS thread must be locked. Otherwise, the exit event of
|
|
// parent process's OS thread will send kill signal to the child process,
|
|
// even if parent process is still running.
|
|
//
|
|
//go:norace
|
|
//go:noinline
|
|
func ForkUserns() (_pid uintptr, _pidfd uintptr, _ syscall.Errno) {
|
|
var (
|
|
pidfd uintptr
|
|
pid, ppid uintptr
|
|
err syscall.Errno
|
|
)
|
|
|
|
ppid, _, err = syscall.RawSyscall(uintptr(syscall.SYS_GETPID), 0, 0, 0)
|
|
if err != 0 {
|
|
return 0, 0, err
|
|
}
|
|
|
|
beforeFork()
|
|
if runtime.GOARCH == "s390x" {
|
|
// NOTE:
|
|
//
|
|
// On the s390 architectures, the order of the first two
|
|
// arguments is reversed.
|
|
//
|
|
// REF: https://man7.org/linux/man-pages/man2/clone.2.html
|
|
pid, _, err = syscall.RawSyscall(syscall.SYS_CLONE,
|
|
0,
|
|
uintptr(syscall.CLONE_NEWUSER|syscall.SIGCHLD|syscall.CLONE_PIDFD),
|
|
uintptr(unsafe.Pointer(&pidfd)),
|
|
)
|
|
} else {
|
|
pid, _, err = syscall.RawSyscall(syscall.SYS_CLONE,
|
|
uintptr(syscall.CLONE_NEWUSER|syscall.SIGCHLD|syscall.CLONE_PIDFD),
|
|
0,
|
|
uintptr(unsafe.Pointer(&pidfd)),
|
|
)
|
|
}
|
|
if err != 0 || pid != 0 {
|
|
afterFork()
|
|
return pid, pidfd, err
|
|
}
|
|
afterForkInChild()
|
|
|
|
if _, _, err = syscall.RawSyscall(syscall.SYS_PRCTL, syscall.PR_SET_PDEATHSIG, uintptr(syscall.SIGKILL), 0); err != 0 {
|
|
goto err
|
|
}
|
|
|
|
pid, _, err = syscall.RawSyscall(syscall.SYS_GETPPID, 0, 0, 0)
|
|
if err != 0 {
|
|
goto err
|
|
}
|
|
|
|
// exit if re-parent
|
|
if pid != ppid {
|
|
goto err
|
|
}
|
|
|
|
_, _, err = syscall.RawSyscall(syscall.SYS_PPOLL, 0, 0, 0)
|
|
err:
|
|
syscall.RawSyscall(syscall.SYS_EXIT, uintptr(err), 0, 0)
|
|
panic("unreachable")
|
|
}
|