Background:
With current design, the content backend uses key-lock for long-lived
write transaction. If the content reference has been marked for write
transaction, the other requestes on the same reference will fail fast with
unavailable error. Since the metadata plugin is based on boltbd which
only supports single-writer, the content backend can't block or handle
the request too long. It requires the client to handle retry by itself,
like OpenWriter - backoff retry helper. But the maximum retry interval
can be up to 2 seconds. If there are several concurrent requestes fo the
same image, the waiters maybe wakeup at the same time and there is only
one waiter can continue. A lot of waiters will get into sleep and we will
take long time to finish all the pulling jobs and be worse if the image
has many more layers, which mentioned in issue #4937.
After fetching, containerd.Pull API allows several hanlers to commit
same ChainID snapshotter but only one can be done successfully. Since
unpack tar.gz is time-consuming job, it can impact the performance on
unpacking for same ChainID snapshotter in parallel.
For instance, the Request 2 doesn't need to prepare and commit, it
should just wait for Request 1 finish, which mentioned in pull
request #6318.
```text
Request 1 Request 2
Prepare
|
|
|
| Prepare
Commit |
|
|
|
Commit(failed on exist)
```
Both content backoff retry and unnecessary unpack impacts the performance.
Solution:
Introduced the duplicate suppression in fetch and unpack context. The
deplicate suppression uses key-mutex and single-waiter-notify to support
singleflight. The caller can use the duplicate suppression in different
PullImage handlers so that we can avoid unnecessary unpack and spin-lock
in OpenWriter.
Test Result:
Before enhancement:
```bash
➜ /tmp sudo bash testing.sh "localhost:5000/redis:latest" 20
crictl pull localhost:5000/redis:latest (x20) takes ...
real 1m6.172s
user 0m0.268s
sys 0m0.193s
docker pull localhost:5000/redis:latest (x20) takes ...
real 0m1.324s
user 0m0.441s
sys 0m0.316s
➜ /tmp sudo bash testing.sh "localhost:5000/golang:latest" 20
crictl pull localhost:5000/golang:latest (x20) takes ...
real 1m47.657s
user 0m0.284s
sys 0m0.224s
docker pull localhost:5000/golang:latest (x20) takes ...
real 0m6.381s
user 0m0.488s
sys 0m0.358s
```
With this enhancement:
```bash
➜ /tmp sudo bash testing.sh "localhost:5000/redis:latest" 20
crictl pull localhost:5000/redis:latest (x20) takes ...
real 0m1.140s
user 0m0.243s
sys 0m0.178s
docker pull localhost:5000/redis:latest (x20) takes ...
real 0m1.239s
user 0m0.463s
sys 0m0.275s
➜ /tmp sudo bash testing.sh "localhost:5000/golang:latest" 20
crictl pull localhost:5000/golang:latest (x20) takes ...
real 0m5.546s
user 0m0.217s
sys 0m0.219s
docker pull localhost:5000/golang:latest (x20) takes ...
real 0m6.090s
user 0m0.501s
sys 0m0.331s
```
Test Script:
localhost:5000/{redis|golang}:latest is equal to
docker.io/library/{redis|golang}:latest. The image is hold in local registry
service by `docker run -d -p 5000:5000 --name registry registry:2`.
```bash
image_name="${1}"
pull_times="${2:-10}"
cleanup() {
ctr image rmi "${image_name}"
ctr -n k8s.io image rmi "${image_name}"
crictl rmi "${image_name}"
docker rmi "${image_name}"
sleep 2
}
crictl_testing() {
for idx in $(seq 1 ${pull_times}); do
crictl pull "${image_name}" > /dev/null 2>&1 &
done
wait
}
docker_testing() {
for idx in $(seq 1 ${pull_times}); do
docker pull "${image_name}" > /dev/null 2>&1 &
done
wait
}
cleanup > /dev/null 2>&1
echo 3 > /proc/sys/vm/drop_caches
sleep 3
echo "crictl pull $image_name (x${pull_times}) takes ..."
time crictl_testing
echo
echo 3 > /proc/sys/vm/drop_caches
sleep 3
echo "docker pull $image_name (x${pull_times}) takes ..."
time docker_testing
```
Fixes: #4937Close: #4985Close: #6318
Signed-off-by: Wei Fu <fuweid89@gmail.com>
The tty test fails on ltsc2022. Disable that test until we manage to
reproduce and fix it.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
To clarify that end users do not need to read these docs, and that these
docs do not need to be updated
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
A bbolt database has a freelist to track all pages that are available
for allocation. However writing the list takes some time and reading
the list sometimes panics.
This commit sets NoFreelistSync true to skipping the freelist entirely,
following what etcd does.
https://github.com/etcd-io/etcd/blob/v3.5.2/server/mvcc/backend/config_linux.go#L31Fixes#4838.
Signed-off-by: Kazuyoshi Kato <katokazu@amazon.com>
This script is used by various tests to install and configure
containerd. However, right now it's hardcoded to only support x86
and not other architectures like arm. This change will now check the
architecture of the machine the script is running on and will pull
the correct artifacts accordingly from the correct artifact store
(GCS vs Github).
Signed-off-by: Nobel Barakat <nobelbarakat@google.com>
The previous documentation was too much forcusing on the Go API and not useful
for users who are not interested in implementing their own containerd client.
It was also recommending the deprecated way (cri-containerd-*.tar.gz) to install
containerd and its dependencies.
The new documentation recommends the current official way to install containerd,
and provides several links for end users.
This will replace the content of https://containerd.io/docs/getting-started/
after merging the containerd/containerd.io PR 120.
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
This change disables Windows Defender real-time monitoring on the test
workers, and increases the test timeout to 20 minutes (default is 10).
The Windows Defender real time monitoring feature scans any newly
created files for malitious contents. This takes up a lot of CPU when
expanding image archives, which contain lots of files. The CI has been
timing out due to the fact that tests take longer than 10 minutes. This
change should address that issue.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
Testing containerd on an EL8 variant will be beneficial for enterprise users.
EL9 is coming soon, but we should keep maintaining EL8 CI for a couple of years for long-time stability.
Fixes issue 6542
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
https://github.com/containers/crun/compare/1.3...1.4.4
Also adds `crun-version` file for consistency with `runc-version`.
(Note: unlike runc, crun does not prepend "v" to a version tag)
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
