github/containerd
4
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 2 Wiki Activity

Merge containerd/cri into containerd/containerd

Browse Source 
Signed-off-by: Derek McGowan <derek@mcg.dev>
This commit is contained in:
Derek McGowan
2020-10-07 10:58:39 -07:00
parent d852786705 0820015314
commit e7a350176a
272 changed files with 35258 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

18
docs/architecture.md Normal file
View File

@@ -0,0 +1,18 @@
# Architecture of The CRI Plugin
This document describes the architecture of the `cri` plugin for `containerd`.
This plugin is an implementation of Kubernetes [container runtime interface (CRI)](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/apis/cri/runtime/v1alpha2/api.proto). Containerd operates on the same node as the [Kubelet](https://kubernetes.io/docs/reference/generated/kubelet/). The `cri` plugin inside containerd handles all CRI service requests from the Kubelet and uses containerd internals to manage containers and container images.
The `cri` plugin uses containerd to manage the full container lifecycle and all container images. As also shown below, `cri` manages pod networking via [CNI](https://github.com/containernetworking/cni) (another CNCF project).
![architecture](./architecture.png)
Let's use an example to demonstrate how the `cri` plugin works for the case when Kubelet creates a single-container pod:
* Kubelet calls the `cri` plugin, via the CRI runtime service API, to create a pod;
* `cri` creates and configures the pod’s network namespace using CNI;
* `cri` uses containerd internal to create and start a special [pause container](https://www.ianlewis.org/en/almighty-pause-container) (the sandbox container) and put that container inside the pod’s cgroups and namespace (steps omitted for brevity);
* Kubelet subsequently calls the `cri` plugin, via the CRI image service API, to pull the application container image;
* `cri` further uses containerd to pull the image if the image is not present on the node;
* Kubelet then calls `cri`, via the CRI runtime service API, to create and start the application container inside the pod using the pulled container image;
* `cri` finally uses containerd internal to create the application container, put it inside the pod’s cgroups and namespace, then to start the pod’s new application container.
After these steps, a pod and its corresponding application container is created and running.

BIN
docs/architecture.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 124 KiB

316
docs/config.md Normal file
View File

@@ -0,0 +1,316 @@
# CRI Plugin Config Guide
This document provides the description of the CRI plugin configuration.
The CRI plugin config is part of the containerd config (default
path: `/etc/containerd/config.toml`).
See [here](https://github.com/containerd/containerd/blob/master/docs/ops.md)
for more information about containerd config.
The explanation and default value of each configuration item are as follows:
```toml
# Use config version 2 to enable new configuration fields.
# Config file is parsed as version 1 by default.
# Version 2 uses long plugin names, i.e. "io.containerd.grpc.v1.cri" vs "cri".
version = 2
# The 'plugins."io.containerd.grpc.v1.cri"' table contains all of the server options.
[plugins."io.containerd.grpc.v1.cri"]
# disable_tcp_service disables serving CRI on the TCP server.
# Note that a TCP server is enabled for containerd if TCPAddress is set in section [grpc].
disable_tcp_service = true
# stream_server_address is the ip address streaming server is listening on.
stream_server_address = "127.0.0.1"
# stream_server_port is the port streaming server is listening on.
stream_server_port = "0"
# stream_idle_timeout is the maximum time a streaming connection can be
# idle before the connection is automatically closed.
# The string is in the golang duration format, see:
# https://golang.org/pkg/time/#ParseDuration
stream_idle_timeout = "4h"
# enable_selinux indicates to enable the selinux support.
enable_selinux = false
# selinux_category_range allows the upper bound on the category range to be set.
# if not specified or set to 0, defaults to 1024 from the selinux package.
selinux_category_range = 1024
# sandbox_image is the image used by sandbox container.
sandbox_image = "k8s.gcr.io/pause:3.2"
# stats_collect_period is the period (in seconds) of snapshots stats collection.
stats_collect_period = 10
# enable_tls_streaming enables the TLS streaming support.
# It generates a self-sign certificate unless the following x509_key_pair_streaming are both set.
enable_tls_streaming = false
# tolerate_missing_hugetlb_controller if set to false will error out on create/update
# container requests with huge page limits if the cgroup controller for hugepages is not present.
# This helps with supporting Kubernetes <=1.18 out of the box. (default is `true`)
tolerate_missing_hugetlb_controller = true
# ignore_image_defined_volumes ignores volumes defined by the image. Useful for better resource
# isolation, security and early detection of issues in the mount configuration when using
# ReadOnlyRootFilesystem since containers won't silently mount a temporary volume.
ignore_image_defined_volumes = false
# 'plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming' contains a x509 valid key pair to stream with tls.
[plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
# tls_cert_file is the filepath to the certificate paired with the "tls_key_file"
tls_cert_file = ""
# tls_key_file is the filepath to the private key paired with the "tls_cert_file"
tls_key_file = ""
# max_container_log_line_size is the maximum log line size in bytes for a container.
# Log line longer than the limit will be split into multiple lines. -1 means no
# limit.
max_container_log_line_size = 16384
# disable_cgroup indicates to disable the cgroup support.
# This is useful when the daemon does not have permission to access cgroup.
disable_cgroup = false
# disable_apparmor indicates to disable the apparmor support.
# This is useful when the daemon does not have permission to access apparmor.
disable_apparmor = false
# restrict_oom_score_adj indicates to limit the lower bound of OOMScoreAdj to
# the containerd's current OOMScoreAdj.
# This is useful when the containerd does not have permission to decrease OOMScoreAdj.
restrict_oom_score_adj = false
# max_concurrent_downloads restricts the number of concurrent downloads for each image.
max_concurrent_downloads = 3
# disable_proc_mount disables Kubernetes ProcMount support. This MUST be set to `true`
# when using containerd with Kubernetes <=1.11.
disable_proc_mount = false
# unsetSeccompProfile is the profile containerd/cri will use if the provided seccomp profile is
# unset (`""`) for a container (default is `unconfined`)
unset_seccomp_profile = ""
# 'plugins."io.containerd.grpc.v1.cri".containerd' contains config related to containerd
[plugins."io.containerd.grpc.v1.cri".containerd]
# snapshotter is the snapshotter used by containerd.
snapshotter = "overlayfs"
# no_pivot disables pivot-root (linux only), required when running a container in a RamDisk with runc.
# This only works for runtime type "io.containerd.runtime.v1.linux".
no_pivot = false
# disable_snapshot_annotations disables to pass additional annotations (image
# related information) to snapshotters. These annotations are required by
# stargz snapshotter (https://github.com/containerd/stargz-snapshotter)
disable_snapshot_annotations = false
# discard_unpacked_layers allows GC to remove layers from the content store after
# successfully unpacking these layers to the snapshotter.
discard_unpacked_layers = false
# default_runtime_name is the default runtime name to use.
default_runtime_name = "runc"
# 'plugins."io.containerd.grpc.v1.cri".containerd.default_runtime' is the runtime to use in containerd.
# DEPRECATED: use `default_runtime_name` and `plugins."io.containerd.grpc.v1.cri".runtimes` instead.
# Remove in containerd 1.4.
[plugins."io.containerd.grpc.v1.cri".containerd.default_runtime]
# 'plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime' is a runtime to run untrusted workloads on it.
# DEPRECATED: use `untrusted` runtime in `plugins."io.containerd.grpc.v1.cri".runtimes` instead.
# Remove in containerd 1.4.
[plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime]
# 'plugins."io.containerd.grpc.v1.cri".containerd.runtimes' is a map from CRI RuntimeHandler strings, which specify types
# of runtime configurations, to the matching configurations.
# In this example, 'runc' is the RuntimeHandler string to match.
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
# runtime_type is the runtime type to use in containerd.
# The default value is "io.containerd.runc.v2" since containerd 1.4.
# The default value was "io.containerd.runc.v1" in containerd 1.3, "io.containerd.runtime.v1.linux" in prior releases.
runtime_type = "io.containerd.runc.v2"
# pod_annotations is a list of pod annotations passed to both pod
# sandbox as well as container OCI annotations. Pod_annotations also
# supports golang path match pattern - https://golang.org/pkg/path/#Match.
# e.g. ["runc.com.*"], ["*.runc.com"], ["runc.com/*"].
#
# For the naming convention of annotation keys, please reference:
# * Kubernetes: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/#syntax-and-character-set
# * OCI: https://github.com/opencontainers/image-spec/blob/master/annotations.md
pod_annotations = []
# container_annotations is a list of container annotations passed through to the OCI config of the containers.
# Container annotations in CRI are usually generated by other Kubernetes node components (i.e., not users).
# Currently, only device plugins populate the annotations.
container_annotations = []
# privileged_without_host_devices allows overloading the default behaviour of passing host
# devices through to privileged containers. This is useful when using a runtime where it does
# not make sense to pass host devices to the container when privileged. Defaults to false -
# i.e pass host devices through to privileged containers.
privileged_without_host_devices = false
# base_runtime_spec is a file path to a JSON file with the OCI spec that will be used as the base spec that all
# container's are created from.
# Use containerd's `ctr oci spec > /etc/containerd/cri-base.json` to output initial spec file.
# Spec files are loaded at launch, so containerd daemon must be restared on any changes to refresh default specs.
# Still running containers and restarted containers will still be using the original spec from which that container was created.
base_runtime_spec = ""
# 'plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options' is options specific to
# "io.containerd.runc.v1" and "io.containerd.runc.v2". Its corresponding options type is:
# https://github.com/containerd/containerd/blob/v1.3.2/runtime/v2/runc/options/oci.pb.go#L26 .
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
# NoPivotRoot disables pivot root when creating a container.
NoPivotRoot = false
# NoNewKeyring disables new keyring for the container.
NoNewKeyring = false
# ShimCgroup places the shim in a cgroup.
ShimCgroup = ""
# IoUid sets the I/O's pipes uid.
IoUid = 0
# IoGid sets the I/O's pipes gid.
IoGid = 0
# BinaryName is the binary name of the runc binary.
BinaryName = ""
# Root is the runc root directory.
Root = ""
# CriuPath is the criu binary path.
CriuPath = ""
# SystemdCgroup enables systemd cgroups.
SystemdCgroup = false
# CriuImagePath is the criu image path
CriuImagePath = ""
# CriuWorkPath is the criu work path.
CriuWorkPath = ""
# 'plugins."io.containerd.grpc.v1.cri".cni' contains config related to cni
[plugins."io.containerd.grpc.v1.cri".cni]
# bin_dir is the directory in which the binaries for the plugin is kept.
bin_dir = "/opt/cni/bin"
# conf_dir is the directory in which the admin places a CNI conf.
conf_dir = "/etc/cni/net.d"
# max_conf_num specifies the maximum number of CNI plugin config files to
# load from the CNI config directory. By default, only 1 CNI plugin config
# file will be loaded. If you want to load multiple CNI plugin config files
# set max_conf_num to the number desired. Setting max_config_num to 0 is
# interpreted as no limit is desired and will result in all CNI plugin
# config files being loaded from the CNI config directory.
max_conf_num = 1
# conf_template is the file path of golang template used to generate
# cni config.
# If this is set, containerd will generate a cni config file from the
# template. Otherwise, containerd will wait for the system admin or cni
# daemon to drop the config file into the conf_dir.
# This is a temporary backward-compatible solution for kubenet users
# who don't have a cni daemonset in production yet.
# This will be deprecated when kubenet is deprecated.
# See the "CNI Config Template" section for more details.
conf_template = ""
# 'plugins."io.containerd.grpc.v1.cri".registry' contains config related to the registry
[plugins."io.containerd.grpc.v1.cri".registry]
# 'plugins."io.containerd.grpc.v1.cri.registry.headers sets the http request headers to send for all registry requests
[plugins."io.containerd.grpc.v1.cri".registry.headers]
Foo = ["bar"]
# 'plugins."io.containerd.grpc.v1.cri".registry.mirrors' are namespace to mirror mapping for all namespaces.
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://registry-1.docker.io", ]
# 'plugins."io.containerd.grpc.v1.cri".image_decryption' contains config related
# to handling decryption of encrypted container images.
[plugins."io.containerd.grpc.v1.cri".image_decryption]
# key_model defines the name of the key model used for how the cri obtains
# keys used for decryption of encrypted container images.
# The [decryption document](https://github.com/containerd/cri/blob/master/docs/decryption.md)
# contains additional information about the key models available.
#
# Set of available string options: {"", "node"}
# Omission of this field defaults to the empty string "", which indicates no key model,
# disabling image decryption.
#
# In order to use the decryption feature, additional configurations must be made.
# The [decryption document](https://github.com/containerd/cri/blob/master/docs/decryption.md)
# provides information of how to set up stream processors and the containerd imgcrypt decoder
# with the appropriate key models.
#
# Additional information:
# * Stream processors: https://github.com/containerd/containerd/blob/master/docs/stream_processors.md
# * Containerd imgcrypt: https://github.com/containerd/imgcrypt
key_model = "node"
```
## Untrusted Workload
The recommended way to run untrusted workload is to use
[`RuntimeClass`](https://kubernetes.io/docs/concepts/containers/runtime-class/) api
introduced in Kubernetes 1.12 to select RuntimeHandlers configured to run
untrusted workload in `plugins."io.containerd.grpc.v1.cri".containerd.runtimes`.
However, if you are using the legacy `io.kubernetes.cri.untrusted-workload`pod annotation
to request a pod be run using a runtime for untrusted workloads, the RuntimeHandler
`plugins."io.containerd.grpc.v1.cri"cri.containerd.runtimes.untrusted` must be defined first.
When the annotation `io.kubernetes.cri.untrusted-workload` is set to `true` the `untrusted`
runtime will be used. For example, see
[Create an untrusted pod using Kata Containers](https://github.com/kata-containers/documentation/blob/master/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md#create-an-untrusted-pod-using-kata-containers).
## CNI Config Template
Ideally the cni config should be placed by system admin or cni daemon like calico,
weaveworks etc. However, there are still users using [kubenet](https://kubernetes.io/docs/concepts/cluster-administration/network-plugins/#kubenet)
today, who don't have a cni daemonset in production. The cni config template is
a temporary backward-compatible solution for them. This is expected to be
deprecated when kubenet is deprecated.
The cni config template uses the [golang
template](https://golang.org/pkg/text/template/) format. Currently supported
values are:
* `.PodCIDR` is a string of the first CIDR assigned to the node.
* `.PodCIDRRanges` is a string array of all CIDRs assigned to the node. It is
usually used for
[dualstack](https://github.com/kubernetes/enhancements/blob/master/keps/sig-network/20180612-ipv4-ipv6-dual-stack.md) support.
* `.Routes` is a string array of all routes needed. It is usually used for
dualstack support or single stack but IPv4 or IPv6 is decided at runtime.
The [golang template actions](https://golang.org/pkg/text/template/#hdr-Actions)
can be used to render the cni config. For example, you can use the following
template to add CIDRs and routes for dualstack in the CNI config:
```
"ipam": {
"type": "host-local",
"ranges": [{{range $i, $range := .PodCIDRRanges}}{{if $i}}, {{end}}[{"subnet": "{{$range}}"}]{{end}}],
"routes": [{{range $i, $route := .Routes}}{{if $i}}, {{end}}{"dst": "{{$route}}"}{{end}}]
}
```
## Deprecation
The config options of the CRI plugin follow the [Kubernetes deprecation
policy of "admin-facing CLI components"](https://kubernetes.io/docs/reference/using-api/deprecation-policy/#deprecating-a-flag-or-cli).
In summary, when a config option is announced to be deprecated:
* It is kept functional for 6 months or 1 release (whichever is longer);
* A warning is emitted when it is used.

BIN
docs/containerd.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 52 KiB

BIN
docs/cri.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 80 KiB

216
docs/crictl.md Normal file
View File

@@ -0,0 +1,216 @@
CRICTL User Guide
=================
This document presumes you already have `containerd` with the `cri` plugin installed and running.
This document is for developers who wish to debug, inspect, and manage their pods,
containers, and container images.
Before generating issues against this document, `containerd`, `containerd/cri`,
or `crictl` please make sure the issue has not already been submitted.
## Install crictl
If you have not already installed crictl please install the version compatible
with the `cri` plugin you are using. If you are a user, your deployment
should have installed crictl for you. If not, get it from your release tarball.
If you are a developer the current version of crictl is specified [here](../hack/utils.sh).
A helper command has been included to install the dependencies at the right version:
```console
$ make install.deps
```
* Note: The file named `/etc/crictl.yaml` is used to configure crictl
so you don't have to repeatedly specify the runtime sock used to connect crictl
to the container runtime:
```console
$ cat /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: true
```
## Download and Inspect a Container Image
The pull command tells the container runtime to download a container image from
a container registry.
```console
$ crictl pull busybox
...
$ crictl inspecti busybox
... displays information about the image.
```
***Note:*** If you get an error similar to the following when running a `crictl`
command (and your containerd instance is already running):
```console
crictl info
FATA[0000] getting status of runtime failed: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
```
This could be that you are using an incorrect containerd configuration (maybe
from a Docker install). You will need to update your containerd configuration
to the containerd instance that you are running. One way of doing this is as
follows:
```console
$ mv /etc/containerd/config.toml /etc/containerd/config.bak
$ containerd config default > /etc/containerd/config.toml
```
## Directly Load a Container Image
Another way to load an image into the container runtime is with the load
command. With the load command you inject a container image into the container
runtime from a file. First you need to create a container image tarball. For
example to create an image tarball for a pause container using Docker:
```console
$ docker pull k8s.gcr.io/pause-amd64:3.2
3.2: Pulling from pause-amd64
67ddbfb20a22: Pull complete
Digest: sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610
Status: Downloaded newer image for k8s.gcr.io/pause-amd64:3.2
$ docker save k8s.gcr.io/pause-amd64:3.2 -o pause.tar
```
Then use [`ctr`](https://github.com/containerd/containerd/blob/master/docs/man/ctr.1.md)
to load the container image into the container runtime:
```console
# The cri plugin uses the "k8s.io" containerd namespace.
$ sudo ctr -n=k8s.io images import pause.tar
Loaded image: k8s.gcr.io/pause-amd64:3.2
```
List images and inspect the pause image:
```console
$ sudo crictl images
IMAGE TAG IMAGE ID SIZE
docker.io/library/busybox latest f6e427c148a76 728kB
k8s.gcr.io/pause-amd64 3.2 da86e6ba6ca19 746kB
$ sudo crictl inspecti da86e6ba6ca19
... displays information about the pause image.
$ sudo crictl inspecti k8s.gcr.io/pause-amd64:3.2
... displays information about the pause image.
```
## Run a pod sandbox (using a config file)
```console
$ cat sandbox-config.json
{
"metadata": {
"name": "nginx-sandbox",
"namespace": "default",
"attempt": 1,
"uid": "hdishd83djaidwnduwk28bcsb"
},
"linux": {
}
}
$ crictl runp sandbox-config.json
e1c83b0b8d481d4af8ba98d5f7812577fc175a37b10dc824335951f52addbb4e
$ crictl pods
PODSANDBOX ID CREATED STATE NAME NAMESPACE ATTEMPT
e1c83b0b8d481 2 hours ago SANDBOX_READY nginx-sandbox default 1
$ crictl inspectp e1c8
... displays information about the pod and the pod sandbox pause container.
```
* Note: As shown above, you may use truncated IDs if they are unique.
* Other commands to manage the pod include `stops ID` to stop a running pod and
`rmp ID` to remove a pod sandbox.
## Create and Run a Container in the Pod Sandbox (using a config file)
```console
$ cat container-config.json
{
"metadata": {
"name": "busybox"
},
"image":{
"image": "busybox"
},
"command": [
"top"
],
"linux": {
}
}
$ crictl create e1c83 container-config.json sandbox-config.json
0a2c761303163f2acaaeaee07d2ba143ee4cea7e3bde3d32190e2a36525c8a05
$ crictl ps -a
CONTAINER ID IMAGE CREATED STATE NAME ATTEMPT
0a2c761303163 docker.io/busybox 2 hours ago CONTAINER_CREATED busybox 0
$ crictl start 0a2c
0a2c761303163f2acaaeaee07d2ba143ee4cea7e3bde3d32190e2a36525c8a05
$ crictl ps
CONTAINER ID IMAGE CREATED STATE NAME ATTEMPT
0a2c761303163 docker.io/busybox 2 hours ago CONTAINER_RUNNING busybox 0
$ crictl inspect 0a2c7
... show detailed information about the container
```
## Exec a Command in the Container
```console
$ crictl exec -i -t 0a2c ls
bin dev etc home proc root sys tmp usr var
```
## Display Stats for the Container
```console
$ crictl stats
CONTAINER CPU % MEM DISK INODES
0a2c761303163f 0.00 983kB 16.38kB 6
```
* Other commands to manage the container include `stop ID` to stop a running
container and `rm ID` to remove a container.
## Display Version Information
```console
$ crictl version
Version: 0.1.0
RuntimeName: containerd
RuntimeVersion: 1.0.0-beta.1-186-gdd47a72-TEST
RuntimeApiVersion: v1alpha2
```
## Display Status & Configuration Information about Containerd & The CRI Plugin
```console
$ crictl info
{
"status": {
"conditions": [
{
"type": "RuntimeReady",
"status": true,
"reason": "",
"message": ""
},
{
"type": "NetworkReady",
"status": true,
"reason": "",
"message": ""
}
]
},
"config": {
"containerd": {
"snapshotter": "overlayfs",
"runtime": "io.containerd.runtime.v1.linux"
},
"cni": {
"binDir": "/opt/cni/bin",
"confDir": "/etc/cni/net.d"
},
"registry": {
"mirrors": {
"docker.io": {
"endpoint": [
"https://registry-1.docker.io"
]
}
}
},
"streamServerPort": "10010",
"sandboxImage": "k8s.gcr.io/pause:3.2",
"statsCollectPeriod": 10,
"containerdRootDir": "/var/lib/containerd",
"containerdEndpoint": "unix:///run/containerd/containerd.sock",
"rootDir": "/var/lib/containerd/io.containerd.grpc.v1.cri",
"stateDir": "/run/containerd/io.containerd.grpc.v1.cri",
},
"golang": "go1.10"
}
```
## More Information
See [here](https://github.com/kubernetes-sigs/cri-tools/blob/master/docs/crictl.md)
for information about crictl.

46
docs/decryption.md Normal file
View File

@@ -0,0 +1,46 @@
# Configure Image Decryption
This document describes the method to configure encrypted container image decryption for `containerd` for use with the `cri` plugin.
## Encrypted Container Images
Encrypted container images are OCI images which contain encrypted blobs. These encrypted images can be created through the use of [containerd/imgcrypt project](https://github.com/containerd/imgcrypt). To decrypt these images, the `containerd` runtime uses information passed from the `cri` such as keys, options and encryption metadata.
## The "node" Key Model
Encryption ties trust to an entity based on the model in which a key is associated with it. We call this the key model. One such usecase is when we want to tie the trust of a key to the node in a cluster. In this case, we call it the "node" or "host" Key Model. Future work will include more key models to facilitate other trust associations (i.e. for multi-tenancy).
### "node" Key Model Usecase
In this model encryption is tied to worker nodes. The usecase here revolves around the idea that an image should be decryptable only on trusted host. Using this model, various node based technologies which help bootstrap trust in worker nodes and perform secure key distribution (i.e. TPM, host attestation, secure/measured boot). In this scenario, runtimes are capable of fetching the necessary decryption keys. An example of this is using the [`--decryption-keys-path` flag in imgcrypt](https://github.com/containerd/imgcrypt).
### Configuring image decryption for "node" key model
The default configuration does not handle decrypting encrypted container images.
An example for configuring the "node" key model for container image decryption:
Configure `cri` to enable decryption with "node" key model
```toml
[plugins."io.containerd.grpc.v1.cri".image_decryption]
key_model = "node"
```
Configure `containerd` daemon [`stream_processors`](https://github.com/containerd/containerd/blob/master/docs/stream_processors.md) to handle the
encrypted mediatypes.
```toml
[stream_processors]
[stream_processors."io.containerd.ocicrypt.decoder.v1.tar.gzip"]
accepts = ["application/vnd.oci.image.layer.v1.tar+gzip+encrypted"]
returns = "application/vnd.oci.image.layer.v1.tar+gzip"
path = "/usr/local/bin/ctd-decoder"
args = ["--decryption-keys-path", "/keys"]
[stream_processors."io.containerd.ocicrypt.decoder.v1.tar"]
accepts = ["application/vnd.oci.image.layer.v1.tar+encrypted"]
returns = "application/vnd.oci.image.layer.v1.tar"
path = "/usr/local/bin/ctd-decoder"
args = ["--decryption-keys-path", "/keys"]
```
In this example, container image decryption is set to use the "node" key model. In addition, the decryption [`stream_processors`](https://github.com/containerd/containerd/blob/master/docs/stream_processors.md) are configured as specified in [containerd/imgcrypt project](https://github.com/containerd/imgcrypt), with the additional field `--decryption-keys-path` configured to specify where decryption keys are located locally in the node.
After modify this config, you need restart the `containerd` service.

108
docs/installation.md Normal file
View File

@@ -0,0 +1,108 @@
# Install Containerd with Release Tarball
This document provides the steps to install `containerd` and its dependencies with the release tarball, and bring up a Kubernetes cluster using kubeadm.
These steps have been verified on Ubuntu 16.04. For other OS distributions, the steps may differ. Please feel free to file issues or PRs if you encounter any problems on other OS distributions.
*Note: You need to run the following steps on each node you are planning to use in your Kubernetes cluster.*
## Release Tarball
For each `containerd` release, we'll publish a release tarball specifically for Kubernetes named `cri-containerd-${VERSION}.${OS}-${ARCH}.tar.gz`. This release tarball contains all required binaries and files for using `containerd` with Kubernetes. For example, the 1.2.4 version is available at https://storage.googleapis.com/cri-containerd-release/cri-containerd-1.2.4.linux-amd64.tar.gz.
Note: The VERSION tag specified for the tarball corresponds to the `containerd` release tag, not a containerd/cri repository release tag. The `containerd` release includes the containerd/cri repository code through vendoring. The containerd/cri version of the containerd/cri code included in `containerd` is specified via a commit hash for containerd/cri in containerd/containerd/vendor.conf.
### Content
As shown below, the release tarball contains:
1) `containerd`, `containerd-shim`, `containerd-stress`, `containerd-release`, `ctr`: binaries for containerd.
2) `runc`: runc binary.
3) `crictl`, `crictl.yaml`: command line tools for CRI container runtime and its config file.
4) `critest`: binary to run [CRI validation test](https://github.com/kubernetes-sigs/cri-tools/blob/master/docs/validation.md).
5) `containerd.service`: Systemd unit for containerd.
6) `/opt/containerd/cluster/`: scripts for `kube-up.sh`.
```console
$ tar -tf cri-containerd-1.1.0-rc.0.linux-amd64.tar.gz
./
./opt
./opt/containerd
./opt/containerd/cluster
./opt/containerd/cluster/gce
./opt/containerd/cluster/gce/cloud-init
./opt/containerd/cluster/gce/cloud-init/node.yaml
./opt/containerd/cluster/gce/cloud-init/master.yaml
./opt/containerd/cluster/gce/configure.sh
./opt/containerd/cluster/gce/env
./opt/containerd/cluster/version
./opt/containerd/cluster/health-monitor.sh
./usr
./usr/local
./usr/local/sbin
./usr/local/sbin/runc
./usr/local/bin
./usr/local/bin/crictl
./usr/local/bin/containerd
./usr/local/bin/containerd-stress
./usr/local/bin/critest
./usr/local/bin/containerd-release
./usr/local/bin/containerd-shim
./usr/local/bin/ctr
./etc
./etc/systemd
./etc/systemd/system
./etc/systemd/system/containerd.service
./etc/crictl.yaml
```
### Binary Information
Information about the binaries in the release tarball:
| Binary Name | Support | OS | Architecture |
|:------------------------------:|:------------------:|:-----:|:------------:|
| containerd | seccomp, apparmor, <br/> overlay, btrfs | linux | amd64 |
| containerd-shim | overlay, btrfs | linux | amd64 |
| runc | seccomp, apparmor | linux | amd64 |
If you have other requirements for the binaries, e.g. selinux support, another architecture support etc., you need to build the binaries yourself following [the instructions](../README.md#getting-started-for-developers).
### Download
The release tarball could be downloaded from the release GCS bucket https://storage.googleapis.com/cri-containerd-release/.
## Step 0: Install Dependent Libraries
Install required library for seccomp.
```bash
sudo apt-get update
sudo apt-get install libseccomp2
```
Note that:
1) If you are using Ubuntu <=Trusty or Debian <=jessie, a backported version of `libseccomp2` is needed. (See the [trusty-backports](https://packages.ubuntu.com/trusty-backports/libseccomp2) and [jessie-backports](https://packages.debian.org/jessie-backports/libseccomp2)).
## Step 1: Download Release Tarball
Download release tarball for the `containerd` version you want to install from the GCS bucket.
```bash
wget https://storage.googleapis.com/cri-containerd-release/cri-containerd-${VERSION}.linux-amd64.tar.gz
```
Validate checksum of the release tarball:
```bash
sha256sum cri-containerd-${VERSION}.linux-amd64.tar.gz
curl https://storage.googleapis.com/cri-containerd-release/cri-containerd-${VERSION}.linux-amd64.tar.gz.sha256
# Compare to make sure the 2 checksums are the same.
```
## Step 2: Install Containerd
If you are using systemd, just simply unpack the tarball to the root directory:
```bash
sudo tar --no-overwrite-dir -C / -xzf cri-containerd-${VERSION}.linux-amd64.tar.gz
sudo systemctl start containerd
```
If you are not using systemd, please unpack all binaries into a directory in your `PATH`, and start `containerd` as monitored long running services with the service manager you are using e.g. `supervisord`, `upstart` etc.
## Step 3: Install Kubeadm, Kubelet and Kubectl
Follow [the instructions](https://kubernetes.io/docs/setup/independent/install-kubeadm/) to install kubeadm, kubelet and kubectl.
## Step 4: Create Systemd Drop-In for Containerd
Create the systemd drop-in file `/etc/systemd/system/kubelet.service.d/0-containerd.conf`:
```
[Service]
Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock"
```
And reload systemd configuration:
```bash
systemctl daemon-reload
```
## Bring Up the Cluster
Now you should have properly installed all required binaries and dependencies on each of your node.
The next step is to use kubeadm to bring up the Kubernetes cluster. It is the same with [the ansible installer](../contrib/ansible). Please follow the steps 2-4 [here](../contrib/ansible/README.md#step-2).

25
docs/kube-up.md Normal file
View File

@@ -0,0 +1,25 @@
# Production Quality Cluster on GCE
This document provides the steps to bring up a production quality cluster on GCE with [`kube-up.sh`](https://kubernetes.io/docs/setup/turnkey/gce/).
**If your Kubernetes version is 1.15 or greater, you can simply run:**
```
export KUBE_CONTAINER_RUNTIME=containerd
```
Follow these instructions [here](https://kubernetes.io/docs/setup/turnkey/gce/) to create a production quality Kubernetes cluster on GCE.
## Download CRI-Containerd Release Tarball
To download release tarball, see [step 1](./installation.md#step-1-download-cri-containerd-release-tarball) in installation.md.
Unpack release tarball to any directory, using `${CRI_CONTAINERD_PATH}` to indicate the directory in the doc:
```bash
tar -C ${CRI_CONTAINERD_PATH} -xzf cri-containerd-${VERSION}.linux-amd64.tar.gz
```
## Set Environment Variables for CRI-Containerd
```bash
. ${CRI_CONTAINERD_PATH}/opt/containerd/cluster/gce/env
```
## Create Kubernetes Cluster on GCE
Follow these instructions [here](https://kubernetes.io/docs/setup/turnkey/gce/) to create a production quality Kubernetes cluster on GCE.
**Make sure the Kubernetes version you are using is v1.11 or greater:**
* When using `https://get.k8s.io`, use the environment variable `KUBERNETES_RELEASE` to set version.
* When using a Kubernetes release tarball, make sure to select version 1.11 or greater.

BIN
docs/performance.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 69 KiB

111
docs/proposal.md Normal file
View File

@@ -0,0 +1,111 @@
Containerd CRI Integration
=============
Author: Lantao Liu (@random-liu)
## Abstract
This proposal aims to integrate [containerd](https://github.com/containerd/containerd) with Kubelet against the [container runtime interface (CRI)](https://github.com/kubernetes/kubernetes/blob/v1.6.0/pkg/kubelet/api/v1alpha1/runtime/api.proto).
## Background
Containerd is a core container runtime, which provides the minimum set of functionalities to manage the complete container lifecycle of its host system, including container execution and supervision, image distribution and storage, etc.
Containerd was [introduced in Docker 1.11](https://blog.docker.com/2016/04/docker-engine-1-11-runc/), used to manage [runC](https://runc.io/) containers on the node. As shown below, it creates a containerd-shim for each container, and the shim manages the lifecycle of its corresponding container.
![containerd](./containerd.png)
In Dec. 2016, Docker Inc. spun it out into a standalone component, and donated it to [CNCF](https://www.cncf.io/) in Mar. 2017.
## Motivation
Containerd is one potential alternative to Docker as the runtime for Kubernetes clusters. *Compared with Docker*, containerd has pros and cons.
### Pros
* **Stability**: Containerd has limited scope and slower feature velocity, which is expected to be more stable.
* **Compatibility**: The scope of containerd aligns with Kubernetes' requirements. It provides the required functionalities and the flexibility for areas like image pulling, networking, volume and logging etc.
* **Performance**:
* Containerd consumes less resource than Docker at least because it's a subset of Docker;
* Containerd CRI integration eliminates an extra hop in the stack (as shown below). ![performance](./performance.png)
* **Neutral Foundation**: Containerd is part of CNCF now.
### Cons
* **User Adoption**:
* Ideally, Kubernetes users don't interact with the underlying container runtime directly. However, for the lack of debug toolkits, sometimes users still need to login the node to debug with Docker CLI directly.
* Containerd provides barebone CLIs [ctr](https://github.com/containerd/containerd/tree/master/cmd/ctr) and [dist](https://github.com/containerd/containerd/tree/master/cmd/dist) for development and debugging purpose, but they may not be sufficient and necessary. Additionally, presuming these are sufficient and necessary tools, a plan and time would be needed to sufficiently document these CLIs and educate users in their use.
* **Maturity**: The rescoped containerd is pretty new, and it's still under heavy development.
## Goals
* Make sure containerd meets the requirement of Kubernetes, now and into the foreseeable future.
* Implement containerd CRI shim and make sure it provides equivalent functionality, usability and debuggability.
* Improve Kubernetes by taking advantage of the flexibility provided by containerd.
## Design
The following sections discuss the design aspects of the containerd CRI integration. For the purposes of this doc, the containerd CRI integration will be referred to as `CRI-containerd`.
### Container Lifecycle
CRI-containerd relies on containerd to manage container lifecycle.
Ideally, CRI-containerd only needs to do api translation and information reorganization. However, CRI-containerd needs to maintain some metadata because:
* There is a mismatch between container lifecycle of CRI and containerd - containerd only tracks running processes, once the container and it's corresponding containerd-shim exit, the container is no longer visible in the containerd API.
* Some sandbox/container metadata is not provided by containerd, and we can not leverage OCI runtime annotation to store it because of the container lifecycle mismatch, e.g. labels/annotations, `PodSandboxID` of a container, `FinishedAt` timestamp, `ExitCode`, `Mounts` etc.
CRI-containerd should checkpoint these metadata itself or use [containerd metadata service](https://github.com/containerd/containerd/blob/0a5544d8c4dab44dfc682f5ad07f1cd011c0a115/design/plugins.md#core) if available.
### Container Logging
Containerd doesn't provide persistent container log. It redirects container STDIO into different FIFOs.
CRI-containerd should start a goroutine (process/container in the future) to:
* Continuously drain the FIFO;
* Decorate the log line into [CRI-defined format](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/kubelet-cri-logging.md#proposed-solution);
* Write the log into [CRI-defined log path](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/kubelet-cri-logging.md#proposed-solution).
### Container Streaming
Containerd supports creating a process in the container with `Exec`, and the STDIO is also exposed as FIFOs. Containerd also supports resizing console of a specific process with `Pty`.
CRI-containerd could reuse the [streaming server](https://github.com/kubernetes/kubernetes/blob/release-1.6/pkg/kubelet/server/streaming/server.go), it should implement the [streaming runtime interface](https://github.com/kubernetes/kubernetes/blob/release-1.6/pkg/kubelet/server/streaming/server.go#L61-L65).
For different CRI streaming functions:
* `ExecSync`: CRI-containerd should use `Exec` to create the exec process, collect the stdout/stderr of the process, and wait for the process to terminate.
* `Exec`: CRI-containerd should use `Exec` to create the exec process, create a goroutine (process/container) to redirect streams, and wait for the process to terminate.
* `Attach`: CRI-containerd should create a goroutine (process/container) to read the existing container log to the output, redirect streams of the init process, and wait for any stream to be closed.
* `PortForward`: CRI-containerd could implement this with `socat` and `nsenter`, similar with [current Docker portforward implementation](https://github.com/kubernetes/kubernetes/blob/release-1.6/pkg/kubelet/dockertools/docker_manager.go#L1373-L1428).
### Container Networking
Containerd doesn't provide container networking, but OCI runtime spec supports joining a linux container into an existing network namespace.
CRI-containerd should:
* Create a network namespace for a sandbox;
* Call [network plugin](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/network/plugins.go) to update the options of the network namespace;
* Let the user containers in the same sandbox share the network namespace.
### Container Metrics
Containerd provides [container cgroup metrics](https://github.com/containerd/containerd/blob/master/reports/2017-03-17.md#metrics), and plans to provide [container writable layer disk usage](https://github.com/containerd/containerd/issues/678).
CRI container metrics api needs to be defined ([#27097](https://github.com/kubernetes/kubernetes/issues/27097)). After that, CRI-containerd should translate containerd container metrics into CRI container metrics.
### Image Management
CRI-containerd relies on containerd to manage images. Containerd should provide all function and information required by CRI, and CRI-containerd only needs to do api translation and information reorganization.
### ImageFS Metrics
Containerd plans to provide [image filesystem metrics](https://github.com/containerd/containerd/issues/678).
CRI image filesystem metrics needs to be defined ([#33048](https://github.com/kubernetes/kubernetes/issues/33048)). After that, we should make sure containerd provides the required metrics, and CRI-containerd should translate containerd image filesystem metrics into CRI image filesystem metrics.
### Out of Scope
Following items are out of the scope of this design, we may address them in future version as enhancement or optimization.
* **Debuggability**: One of the biggest concern of CRI-containerd is debuggability. We should provide equivalent debuggability with Docker CLI through `kubectl`, [`cri-tools`](https://github.com/kubernetes-sigs/cri-tools) or containerd CLI.
* **Built-in CRI support**: The [plugin model](https://github.com/containerd/containerd/blob/master/design/plugins.md) provided by containerd makes it possible to directly build CRI support into containerd as a plugin, which will eliminate one more hop from the stack. But because of the [limitation of golang plugin](https://github.com/containerd/containerd/issues/563), we have to either maintain our own branch or push CRI plugin upstream.
* **Seccomp**: ([#36997](https://github.com/kubernetes/kubernetes/issues/36997)) Seccomp is supported in OCI runtime spec. However, current seccomp implementation in Kubernetes is experimental and docker specific, the api needs to be defined in CRI first before CRI-containerd implements it.
* **Streaming server authentication**: ([#36666](https://github.com/kubernetes/kubernetes/issues/36666)) CRI-containerd will be out-of-process with Kubelet, so it could not reuse Kubelet authentication. Its streaming server should implement its own authentication mechanism.
* **Move container facilities into pod cgroup**: Container facilities including container image puller, container streaming handler, log handler and containerd-shim serve a specific container. They should be moved to the corresponding pod cgroup, and the overhead introduced by them should be charged to the pod.
* **Log rotation**: ([#42718](https://github.com/kubernetes/kubernetes/issues/42718)) Container log rotation is under design. A function may be added in CRI to signal the runtime to reopen log file. CRI-containerd should implement that function after it is defined.
* **Exec container**: With the flexibility provided by containerd, it is possible to implement `Exec` with a separate container sharing the same rootfs and mount namespace with the original container. The advantage is that the `Exec` container could have it's own sub-cgroup, so that it will not consume the resource of application container and user could specify dedicated resource for it.
* **Advanced image management**: The image management interface in CRI is relatively simple because the requirement of Kubelet image management is not clearly scoped out. In the future, we may want to leverage the flexibility provided by containerd more, e.g. estimate image size before pulling etc.
* ...
## Roadmap and Milestones
### Milestones
#### Kubernetes 1.7 - Q2
* [P0] Basic container lifecycle.
* [P0] Basic image management.
* [P0] Container networking.
* [P1] Container streaming/logging.
* [P2] Container/ImageFS Metrics.
*Test Plan: Each feature added should have unit test and pass its corresponding cri validation test.*
#### Kubernetes 1.8 - Q3
* [P0] Feature complete, pass 100% cri validation test.
* [P0] Integrate CRI-containerd with Kubernetes, and build the e2e/node e2e test framework.
* [P1] Address the debuggability problem.
### Q2 Roadmap
| Item | 1/2 Mar. | 2/2 Mar. | 1/2 Apr. | 2/2 Apr. | 1/2 May. | 2/2 May. |
|:--------------------------------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|
| Survey | ✓ | | | | | |
| POC | | ✓ | | | | |
| Proposal | | | ✓ | | | |
| Containerd Feature Complete | ✓ | ✓ | ✓ | | | |
| Runtime Management Integration | | | ✓ | ✓ | ✓ | ✓ |
| Image Management Integration | | | | ✓ | ✓ | ✓ |
| Container Networking Integration | | | | | ✓ | ✓ |

187
docs/registry.md Normal file
View File

@@ -0,0 +1,187 @@
# Configure Image Registry
This document describes the method to configure the image registry for `containerd` for use with the `cri` plugin.
NOTE: The configuration syntax used in this doc is in version 2 which is the
recommended since `containerd` 1.3. If your configuration is still in version 1,
you can replace `"io.containerd.grpc.v1.cri"` with `cri`.
## Configure Registry Endpoint
With containerd, `docker.io` is the default image registry. You can also set up other image registries similar to docker.
To configure image registries create/modify the `/etc/containerd/config.toml` as follows:
```toml
# Config file is parsed as version 1 by default.
# To use the long form of plugin names set "version = 2"
# explicitly use v2 config format
version = 2
[plugin."io.containerd.grpc.v1.cri".registry.mirrors]
[plugin."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://registry-1.docker.io"]
[plugin."io.containerd.grpc.v1.cri".registry.mirrors."test.https-registry.io"]
endpoint = ["https://HostIP1:Port1"]
[plugin."io.containerd.grpc.v1.cri".registry.mirrors."test.http-registry.io"]
endpoint = ["http://HostIP2:Port2"]
# wildcard matching is supported but not required.
[plugin."io.containerd.grpc.v1.cri".registry.mirrors."*"]
endpoint = ["https://HostIP3:Port3"]
```
The default configuration can be generated by `containerd config default > /etc/containerd/config.toml`.
The endpoint is a list that can contain multiple image registry URLs split by commas. When pulling an image
from a registry, containerd will try these endpoint URLs one by one, and use the first working one. Please note
that if the default registry endpoint is not already specified in the endpoint list, it will be automatically
tried at the end with scheme `https` and path `v2`, e.g. `https://gcr.io/v2` for `gcr.io`.
As an example, for the image `gcr.io/library/busybox:latest`, the endpoints are:
* `gcr.io` is configured: endpoints for `gcr.io` + default endpoint `https://gcr.io/v2`.
* `*` is configured, and `gcr.io` is not: endpoints for `*` + default
endpoint `https://gcr.io/v2`.
* None of above is configured: default endpoint `https://gcr.io/v2`.
After modify this config, you need restart the `containerd` service.
## Configure Registry TLS Communication
`cri` plugin also supports configuring TLS settings when communicating with a registry.
To configure the TLS settings for a specific registry, create/modify the `/etc/containerd/config.toml` as follows:
```toml
# explicitly use v2 config format
version = 2
# The registry host has to be a domain name or IP. Port number is also
# needed if the default HTTPS or HTTP port is not used.
[plugin."io.containerd.grpc.v1.cri".registry.configs."my.custom.registry".tls]
ca_file = "ca.pem"
cert_file = "cert.pem"
key_file = "key.pem"
```
In the config example shown above, TLS mutual authentication will be used for communications with the registry endpoint located at <https://my.custom.registry>.
`ca_file` is file name of the certificate authority (CA) certificate used to authenticate the x509 certificate/key pair specified by the files respectively pointed to by `cert_file` and `key_file`.
`cert_file` and `key_file` are not needed when TLS mutual authentication is unused.
```toml
# explicitly use v2 config format
version = 2
[plugin."io.containerd.grpc.v1.cri".registry.configs."my.custom.registry".tls]
ca_file = "ca.pem"
```
To skip the registry certificate verification:
```toml
# explicitly use v2 config format
version = 2
[plugin."io.containerd.grpc.v1.cri".registry.configs."my.custom.registry".tls]
insecure_skip_verify = true
```
## Configure Registry Credentials
`cri` plugin also supports docker like registry credential config.
To configure a credential for a specific registry, create/modify the
`/etc/containerd/config.toml` as follows:
```toml
# explicitly use v2 config format
version = 2
# The registry host has to be a domain name or IP. Port number is also
# needed if the default HTTPS or HTTP port is not used.
[plugin."io.containerd.grpc.v1.cri".registry.configs."gcr.io".auth]
username = ""
password = ""
auth = ""
identitytoken = ""
```
The meaning of each field is the same with the corresponding field in `.docker/config.json`.
Please note that auth config passed by CRI takes precedence over this config.
The registry credential in this config will only be used when auth config is
not specified by Kubernetes via CRI.
After modifying this config, you need to restart the `containerd` service.
### Configure Registry Credentials Example - GCR with Service Account Key Authentication
If you don't already have Google Container Registry (GCR) set-up then you need to do the following steps:
* Create a Google Cloud Platform (GCP) account and project if not already created (see [GCP getting started](https://cloud.google.com/gcp/getting-started))
* Enable GCR for your project (see [Quickstart for Container Registry](https://cloud.google.com/container-registry/docs/quickstart))
* For authentication to GCR: Create [service account and JSON key](https://cloud.google.com/container-registry/docs/advanced-authentication#json-key)
* The JSON key file needs to be downloaded to your system from the GCP console
* For access to the GCR storage: Add service account to the GCR storage bucket with storage admin access rights (see [Granting permissions](https://cloud.google.com/container-registry/docs/access-control#grant-bucket))
Refer to [Pushing and pulling images](https://cloud.google.com/container-registry/docs/pushing-and-pulling) for detailed information on the above steps.
> Note: The JSON key file is a multi-line file and it can be cumbersome to use the contents as a key outside of the file. It is worthwhile generating a single line format output of the file. One way of doing this is using the `jq` tool as follows: `jq -c . key.json`
It is beneficial to first confirm that from your terminal you can authenticate with your GCR and have access to the storage before hooking it into containerd. This can be verified by performing a login to your GCR and
pushing an image to it as follows:
```console
docker login -u _json_key -p "$(cat key.json)" gcr.io
docker pull busybox
docker tag busybox gcr.io/your-gcp-project-id/busybox
docker push gcr.io/your-gcp-project-id/busybox
docker logout gcr.io
```
Now that you know you can access your GCR from your terminal, it is now time to try out containerd.
Edit the containerd config (default location is at `/etc/containerd/config.toml`)
to add your JSON key for `gcr.io` domain image pull
requests:
```toml
version = 2
[plugins."io.containerd.grpc.v1.cri".registry]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://registry-1.docker.io"]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."gcr.io"]
endpoint = ["https://gcr.io"]
[plugins."io.containerd.grpc.v1.cri".registry.configs]
[plugins."io.containerd.grpc.v1.cri".registry.configs."gcr.io".auth]
username = "_json_key"
password = 'paste output from jq'
```
> Note: `username` of `_json_key` signifies that JSON key authentication will be used.
Restart containerd:
```console
service containerd restart
```
Pull an image from your GCR with `crictl`:
```console
$ sudo crictl pull gcr.io/your-gcp-project-id/busybox
DEBU[0000] get image connection
DEBU[0000] connect using endpoint 'unix:///run/containerd/containerd.sock' with '3s' timeout
DEBU[0000] connected successfully using endpoint: unix:///run/containerd/containerd.sock
DEBU[0000] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:gcr.io/your-gcr-instance-id/busybox,},Auth:nil,SandboxConfig:nil,}
DEBU[0001] PullImageResponse: &PullImageResponse{ImageRef:sha256:78096d0a54788961ca68393e5f8038704b97d8af374249dc5c8faec1b8045e42,}
Image is up to date for sha256:78096d0a54788961ca68393e5f8038704b97d8af374249dc5c8faec1b8045e42
```

27
docs/release.md Normal file
View File

@@ -0,0 +1,27 @@
# Release Process
This document describes how to cut a `cri` plugin release.
## Step 1: Update containerd vendor
Update the version of containerd located in `containerd/cri/vendor.conf`
to the latest version of containerd for the desired branch of containerd,
and make sure all tests in CI pass https://k8s-testgrid.appspot.com/sig-node-containerd.
## Step 2: Cut the release
Draft and tag a new release in https://github.com/containerd/cri/releases.
## Step 3: Update `cri` version in containerd
Push a PR to `containerd/containerd` that updates the version of
`containerd/cri` in `containerd/containerd/vendor.conf` to the newly
tagged release created in Step 2.
## Step 4: Iterate step 1 updating containerd vendor
## Step 5: Publish release tarball for Kubernetes
Publish the release tarball `cri-containerd-${CONTAINERD_VERSION}.${OS}-${ARCH}.tar.gz`
```shell
# Checkout `containerd/cri` to the newly released version.
git checkout ${RELEASE_VERSION}
# Publish the release tarball without cni.
DEPLOY_BUCKET=cri-containerd-release make push TARBALL_PREFIX=cri-containerd OFFICIAL_RELEASE=true VERSION=${CONTAINERD_VERSION}
# Publish the release tarball with cni.
DEPLOY_BUCKET=cri-containerd-release make push TARBALL_PREFIX=cri-containerd-cni OFFICIAL_RELEASE=true INCLUDE_CNI=true VERSION=${CONTAINERD_VERSION}
```
## Step 6: Update release note with release tarball information

58
docs/testing.md Normal file
View File

@@ -0,0 +1,58 @@
CRI Plugin Testing Guide
========================
This document assumes you have already setup the development environment (go, git, `containerd/cri` repo etc.).
Before sending pull requests you should at least make sure your changes have passed code verification, unit, integration and CRI validation tests.
## Code Verification
Code verification includes lint, and code formatting check etc.
* Install tools used by code verification:
```bash
make install.tools
```
***Note:*** Some make actions (like `install.tools`) use the user's `GOPATH` and will otherwise not work when it is not set. Other make actions override it by setting it to a temporary directory for release build and testing purposes.
* Run code verification:
```bash
make verify
```
## Unit Test
Run all unit tests in `containerd/cri` repo.
```bash
make test
```
## Integration Test
Run all integration tests in `containerd/cri` repo.
* [Install dependencies](../README.md#install-dependencies).
* Run integration test:
```bash
make test-integration
```
## CRI Validation Test
[CRI validation test](https://github.com/kubernetes/community/blob/master/contributors/devel/cri-validation.md) is a test framework for validating that a Container Runtime Interface (CRI) implementation such as containerd with the `cri` plugin meets all the requirements necessary to manage pod sandboxes, containers, images etc.
CRI validation test makes it possible to verify CRI conformance of `containerd/cri` without setting up Kubernetes components or running Kubernetes end-to-end tests.
* [Install dependencies](../README.md#install-dependencies).
* Build containerd with the `cri` plugin:
```bash
make
```
* Run CRI validation test:
```bash
make test-cri
```
* Focus or skip specific CRI validation test:
```bash
make test-cri FOCUS=REGEXP_TO_FOCUS SKIP=REGEXP_TO_SKIP
```
[More information](https://github.com/kubernetes-sigs/cri-tools) about CRI validation test.
## Node E2E Test
[Node e2e test](https://github.com/kubernetes/community/blob/master/contributors/devel/e2e-node-tests.md) is a test framework testing Kubernetes node level functionalities such as managing pods, mounting volumes etc. It starts a local cluster with Kubelet and a few other minimum dependencies, and runs node functionality tests against the local cluster.
* [Install dependencies](../README.md#install-dependencies).
* Run node e2e test:
```bash
make test-e2e-node
```
* Focus or skip specific node e2e test:
```bash
make test-e2e-node FOCUS=REGEXP_TO_FOCUS SKIP=REGEXP_TO_SKIP
```
[More information](https://github.com/kubernetes/community/blob/master/contributors/devel/e2e-node-tests.md) about Kubernetes node e2e test.