github/containerd
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 2 Wiki Activity

Merge containerd/cri into containerd/containerd

Browse Source 
Signed-off-by: Derek McGowan <derek@mcg.dev>
This commit is contained in:
Derek McGowan 2020-10-07 10:58:39 -07:00
commit e7a350176a
No known key found for this signature in database
GPG Key ID: F58C5D0A4405ACDB
272 changed files with 35258 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

122
contrib/ansible/README.md Normal file
View File

@ -0,0 +1,122 @@
# Kubernetes Cluster with Containerd
<p align="center">
<img src="https://kubernetes.io/images/favicon.png" width="50" height="50">
<img src="https://containerd.io/img/logos/icon/black/containerd-icon-black.png" width="50" >
</p>
This document provides the steps to bring up a Kubernetes cluster using ansible and kubeadm tools.
### Prerequisites:
- **OS**: Ubuntu 16.04 (will be updated with additional distros after testing)
- **Python**: 2.7+
- **Ansible**: 2.4+
## Step 0:
- Install Ansible on the host where you will provision the cluster. This host may be one of the nodes you plan to include in your cluster. Installation instructions for Ansible are found [here](http://docs.ansible.com/ansible/latest/intro_installation.html).
- Create a hosts file and include the IP addresses of the hosts that need to be provisioned by Ansible.
```console
$ cat hosts
172.31.7.230
172.31.13.159
172.31.1.227
```
- Setup passwordless SSH access from the host where you are running Ansible to all the hosts in the hosts file. The instructions can be found in [here](http://www.linuxproblem.org/art_9.html)
## Step 1:
At this point, the ansible playbook should be able to ssh into the machines in the hosts file.
```console
git clone https://github.com/containerd/cri
cd ./cri/contrib/ansible
ansible-playbook -i hosts cri-containerd.yaml
```
A typical cloud login might have a username and private key file, in which case the following can be used:
```console
ansible-playbook -i hosts -u <username> --private-key <example.pem> cri-containerd.yaml
```
For more options ansible config file (/etc/ansible/ansible.cfg) can be used to set defaults. Please refer to [Ansible options](http://docs.ansible.com/ansible/latest/intro_configuration.html) for advanced ansible configurations.
At the end of this step, you will have the required software installed in the hosts to bringup a kubernetes cluster.
```console
PLAY RECAP ***************************************************************************************************************************************************************
172.31.1.227 : ok=21 changed=7 unreachable=0 failed=0
172.31.13.159 : ok=21 changed=7 unreachable=0 failed=0
172.31.7.230 : ok=21 changed=7 unreachable=0 failed=0
```
## Step 2:
Use [kubeadm](https://kubernetes.io/docs/setup/independent/install-kubeadm/) to bring up a Kubernetes Cluster. Depending on what third-party provider you choose, you might have to set the ```--pod-network-cidr``` to something provider-specific.
Initialize the cluster from one of the nodes (Note: This node will be the master node):
```console
$sudo kubeadm init --skip-preflight-checks
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[init] Using Kubernetes version: v1.7.6
[init] Using Authorization modes: [Node RBAC]
[preflight] Skipping pre-flight checks
[kubeadm] WARNING: starting in 1.8, tokens expire after 24 hours by default (if you require a non-expiring token use --token-ttl 0)
[certificates] Generated CA certificate and key.
[certificates] Generated API server certificate and key.
[certificates] API Server serving cert is signed for DNS names [abhi-k8-ubuntu-1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.31.7.230]
[certificates] Generated API server kubelet client certificate and key.
[certificates] Generated service account token signing key and public key.
[certificates] Generated front-proxy CA certificate and key.
[certificates] Generated front-proxy client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[apiclient] Created API client, waiting for the control plane to become ready
[apiclient] All control plane components are healthy after 42.002391 seconds
[token] Using token: 43a25d.420ff2e06336e4c1
[apiconfig] Created RBAC rules
[addons] Applied essential addon: kube-proxy
[addons] Applied essential addon: kube-dns
Your Kubernetes master has initialized successfully!
To start using your cluster, you need to run (as a regular user):
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
http://kubernetes.io/docs/admin/addons/
You can now join any number of machines by running the following on each node
as root:
kubeadm join --token 43a25d.420ff2e06336e4c1 172.31.7.230:6443
```
## Step 3:
Use kubeadm join to add each of the remaining nodes to your cluster. (Note: Uses token that was generated during cluster init.)
```console
$sudo kubeadm join --token 43a25d.420ff2e06336e4c1 172.31.7.230:6443 --skip-preflight-checks
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[preflight] Skipping pre-flight checks
[discovery] Trying to connect to API Server "172.31.7.230:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://172.31.7.230:6443"
[discovery] Cluster info signature and contents are valid, will use API Server "https://172.31.7.230:6443"
[discovery] Successfully established connection with API Server "172.31.7.230:6443"
[bootstrap] Detected server version: v1.7.6
[bootstrap] The server supports the Certificates API (certificates.k8s.io/v1beta1)
[csr] Created API client to obtain unique certificate for this node, generating keys and certificate signing request
[csr] Received signed certificate from the API server, generating KubeConfig...
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
Node join complete:
* Certificate signing request sent to master and response
received.
* Kubelet informed of new secure connection details.
Run 'kubectl get nodes' on the master to see this machine join.
```
At the end of Step 3 you should have a kubernetes cluster up and running and ready for deployment.
## Step 4:
Please follow the instructions [here](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#pod-network) to deploy CNI network plugins and start a demo app.
We are constantly striving to improve the installer. Please feel free to open issues and provide suggestions to make the installer fast and easy to use. We are open to receiving help in validating and improving the installer on different distros.

66
contrib/ansible/cri-containerd.yaml Normal file
View File

@ -0,0 +1,66 @@
---
- hosts: all
become: true
tasks:
- include_vars: vars/vars.yaml # Contains tasks variables for installer
- include_tasks: tasks/bootstrap_ubuntu.yaml # Contains tasks bootstrap components for ubuntu systems
when: ansible_distribution == "Ubuntu"
- include_tasks: tasks/bootstrap_centos.yaml # Contains tasks bootstrap components for centos systems
when: ansible_distribution == "CentOS"
- include_tasks: tasks/k8s.yaml # Contains tasks kubernetes component installation
- include_tasks: tasks/binaries.yaml # Contains tasks for pulling containerd components
- name: "Create a directory for containerd config"
file: path=/etc/containerd state=directory
- name: "Start Containerd"
systemd: name=containerd daemon_reload=yes state=started enabled=yes
- name: "Load br_netfilter kernel module"
modprobe:
name: br_netfilter
state: present
- name: "Set bridge-nf-call-iptables"
sysctl:
name: net.bridge.bridge-nf-call-iptables
value: 1
- name: "Set ip_forward"
sysctl:
name: net.ipv4.ip_forward
value: 1
- name: "Check kubelet args in kubelet config (Ubuntu)"
shell: grep "^Environment=\"KUBELET_EXTRA_ARGS=" /etc/systemd/system/kubelet.service.d/10-kubeadm.conf || true
register: check_args
when: ansible_distribution == "Ubuntu"
- name: "Add runtime args in kubelet conf (Ubuntu)"
lineinfile:
dest: "/etc/systemd/system/kubelet.service.d/10-kubeadm.conf"
line: "Environment=\"KUBELET_EXTRA_ARGS= --runtime-cgroups=/system.slice/containerd.service --container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock\""
insertafter: '\[Service\]'
when: ansible_distribution == "Ubuntu" and check_args.stdout == ""
- name: "Check kubelet args in kubelet config (CentOS)"
shell: grep "^Environment=\"KUBELET_EXTRA_ARGS=" /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf || true
register: check_args
when: ansible_distribution == "CentOS"
- name: "Add runtime args in kubelet conf (CentOS)"
lineinfile:
dest: "/usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf"
line: "Environment=\"KUBELET_EXTRA_ARGS= --runtime-cgroups=/system.slice/containerd.service --container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock\""
insertafter: '\[Service\]'
when: ansible_distribution == "CentOS" and check_args.stdout == ""
- name: "Start Kubelet"
systemd: name=kubelet daemon_reload=yes state=started enabled=yes
# TODO This needs to be removed once we have consistent concurrent pull results
- name: "Pre-pull pause container image"
shell: |
/usr/local/bin/ctr pull k8s.gcr.io/pause:3.2
/usr/local/bin/crictl --runtime-endpoint unix:///run/containerd/containerd.sock \
pull k8s.gcr.io/pause:3.2

12
contrib/ansible/tasks/binaries.yaml Normal file
View File

@ -0,0 +1,12 @@
---
- name: "Get Containerd"
unarchive:
src: "https://storage.googleapis.com/cri-containerd-release/cri-containerd-{{ containerd_release_version }}.linux-amd64.tar.gz"
dest: "/"
remote_src: yes
- name: "Create a directory for cni binary"
file: path={{ cni_bin_dir }} state=directory
- name: "Create a directory for cni config files"
file: path={{ cni_conf_dir }} state=directory

12
contrib/ansible/tasks/bootstrap_centos.yaml Normal file
View File

@ -0,0 +1,12 @@
---
- name: "Install required packages on CentOS "
yum:
name: "{{ item }}"
state: latest
with_items:
- unzip
- tar
- btrfs-progs
- libseccomp
- util-linux
- libselinux-python

12
contrib/ansible/tasks/bootstrap_ubuntu.yaml Normal file
View File

@ -0,0 +1,12 @@
---
- name: "Install required packages on Ubuntu"
package:
name: "{{ item }}"
state: latest
with_items:
- unzip
- tar
- apt-transport-https
- btrfs-tools
- libseccomp2
- util-linux

52
contrib/ansible/tasks/k8s.yaml Normal file
View File

@ -0,0 +1,52 @@
---
- name: "Add gpg key (Ubuntu)"
apt_key:
url: https://packages.cloud.google.com/apt/doc/apt-key.gpg
state: present
when: ansible_distribution == "Ubuntu"
- name: "Add kubernetes source list (Ubuntu)"
apt_repository:
repo: "deb http://apt.kubernetes.io/ kubernetes-{{ ansible_distribution_release }} main"
state: present
filename: "kubernetes"
when: ansible_distribution == "Ubuntu"
- name: "Update the repository cache (Ubuntu)"
apt:
update_cache: yes
when: ansible_distribution == "Ubuntu"
- name: "Add Kubernetes repository and install gpg key (CentOS)"
yum_repository:
name: kubernetes
description: Kubernetes repository
baseurl: https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
gpgcheck: yes
enabled: yes
repo_gpgcheck: yes
gpgkey:
- https://packages.cloud.google.com/yum/doc/yum-key.gpg
- https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
when: ansible_distribution == "CentOS"
- name: "Disable SELinux (CentOS)"
selinux:
state: disabled
when: ansible_distribution == "CentOS"
- name: "Install kubelet,kubeadm,kubectl (CentOS)"
yum: state=present name={{ item }}
with_items:
- kubelet
- kubeadm
- kubectl
when: ansible_distribution == "CentOS"
- name: "Install kubelet, kubeadm, kubectl (Ubuntu)"
apt: name={{item}} state=installed
with_items:
- kubelet
- kubeadm
- kubectl
when: ansible_distribution == "Ubuntu"

4
contrib/ansible/vars/vars.yaml Normal file
View File

@ -0,0 +1,4 @@
---
containerd_release_version: 1.3.0
cni_bin_dir: /opt/cni/bin/
cni_conf_dir: /etc/cni/net.d/

5
contrib/linuxkit/README.md Normal file
View File

@ -0,0 +1,5 @@
# LinuxKit Kubernetes project
The LinuxKit [`projects/kubernetes`](https://github.com/linuxkit/linuxkit/tree/master/projects/kubernetes) subdirectory contains a project to build master and worker node virtual machines. When built with `KUBE_RUNTIME=cri-containerd` then these images will use `cri-containerd` as their execution backend.
See the [project README](https://github.com/linuxkit/linuxkit/blob/master/projects/kubernetes/README.md).

22
contrib/systemd-units/containerd.service Normal file
View File

@ -0,0 +1,22 @@
[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target
[Service]
ExecStartPre=/sbin/modprobe overlay
ExecStart=/usr/local/bin/containerd
Restart=always
RestartSec=5
Delegate=yes
KillMode=process
OOMScoreAdjust=-999
LimitNOFILE=1048576
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
[Install]
WantedBy=multi-user.target

18
docs/architecture.md Normal file
View File

@ -0,0 +1,18 @@
# Architecture of The CRI Plugin
This document describes the architecture of the `cri` plugin for `containerd`.
This plugin is an implementation of Kubernetes [container runtime interface (CRI)](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/apis/cri/runtime/v1alpha2/api.proto). Containerd operates on the same node as the [Kubelet](https://kubernetes.io/docs/reference/generated/kubelet/). The `cri` plugin inside containerd handles all CRI service requests from the Kubelet and uses containerd internals to manage containers and container images.
The `cri` plugin uses containerd to manage the full container lifecycle and all container images. As also shown below, `cri` manages pod networking via [CNI](https://github.com/containernetworking/cni) (another CNCF project).
![architecture](./architecture.png)
Let's use an example to demonstrate how the `cri` plugin works for the case when Kubelet creates a single-container pod:
* Kubelet calls the `cri` plugin, via the CRI runtime service API, to create a pod;
* `cri` creates and configures the pod’s network namespace using CNI;
* `cri` uses containerd internal to create and start a special [pause container](https://www.ianlewis.org/en/almighty-pause-container) (the sandbox container) and put that container inside the pod’s cgroups and namespace (steps omitted for brevity);
* Kubelet subsequently calls the `cri` plugin, via the CRI image service API, to pull the application container image;
* `cri` further uses containerd to pull the image if the image is not present on the node;
* Kubelet then calls `cri`, via the CRI runtime service API, to create and start the application container inside the pod using the pulled container image;
* `cri` finally uses containerd internal to create the application container, put it inside the pod’s cgroups and namespace, then to start the pod’s new application container.
After these steps, a pod and its corresponding application container is created and running.

BIN
docs/architecture.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 124 KiB

316
docs/config.md Normal file
View File

@ -0,0 +1,316 @@
# CRI Plugin Config Guide
This document provides the description of the CRI plugin configuration.
The CRI plugin config is part of the containerd config (default
path: `/etc/containerd/config.toml`).
See [here](https://github.com/containerd/containerd/blob/master/docs/ops.md)
for more information about containerd config.
The explanation and default value of each configuration item are as follows:
```toml
# Use config version 2 to enable new configuration fields.
# Config file is parsed as version 1 by default.
# Version 2 uses long plugin names, i.e. "io.containerd.grpc.v1.cri" vs "cri".
version = 2
# The 'plugins."io.containerd.grpc.v1.cri"' table contains all of the server options.
[plugins."io.containerd.grpc.v1.cri"]
# disable_tcp_service disables serving CRI on the TCP server.
# Note that a TCP server is enabled for containerd if TCPAddress is set in section [grpc].
disable_tcp_service = true
# stream_server_address is the ip address streaming server is listening on.
stream_server_address = "127.0.0.1"
# stream_server_port is the port streaming server is listening on.
stream_server_port = "0"
# stream_idle_timeout is the maximum time a streaming connection can be
# idle before the connection is automatically closed.
# The string is in the golang duration format, see:
# https://golang.org/pkg/time/#ParseDuration
stream_idle_timeout = "4h"
# enable_selinux indicates to enable the selinux support.
enable_selinux = false
# selinux_category_range allows the upper bound on the category range to be set.
# if not specified or set to 0, defaults to 1024 from the selinux package.
selinux_category_range = 1024
# sandbox_image is the image used by sandbox container.
sandbox_image = "k8s.gcr.io/pause:3.2"
# stats_collect_period is the period (in seconds) of snapshots stats collection.
stats_collect_period = 10
# enable_tls_streaming enables the TLS streaming support.
# It generates a self-sign certificate unless the following x509_key_pair_streaming are both set.
enable_tls_streaming = false
# tolerate_missing_hugetlb_controller if set to false will error out on create/update
# container requests with huge page limits if the cgroup controller for hugepages is not present.
# This helps with supporting Kubernetes <=1.18 out of the box. (default is `true`)
tolerate_missing_hugetlb_controller = true
# ignore_image_defined_volumes ignores volumes defined by the image. Useful for better resource
# isolation, security and early detection of issues in the mount configuration when using
# ReadOnlyRootFilesystem since containers won't silently mount a temporary volume.
ignore_image_defined_volumes = false
# 'plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming' contains a x509 valid key pair to stream with tls.
[plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
# tls_cert_file is the filepath to the certificate paired with the "tls_key_file"
tls_cert_file = ""
# tls_key_file is the filepath to the private key paired with the "tls_cert_file"
tls_key_file = ""
# max_container_log_line_size is the maximum log line size in bytes for a container.
# Log line longer than the limit will be split into multiple lines. -1 means no
# limit.
max_container_log_line_size = 16384
# disable_cgroup indicates to disable the cgroup support.
# This is useful when the daemon does not have permission to access cgroup.
disable_cgroup = false
# disable_apparmor indicates to disable the apparmor support.
# This is useful when the daemon does not have permission to access apparmor.
disable_apparmor = false
# restrict_oom_score_adj indicates to limit the lower bound of OOMScoreAdj to
# the containerd's current OOMScoreAdj.
# This is useful when the containerd does not have permission to decrease OOMScoreAdj.
restrict_oom_score_adj = false
# max_concurrent_downloads restricts the number of concurrent downloads for each image.
max_concurrent_downloads = 3
# disable_proc_mount disables Kubernetes ProcMount support. This MUST be set to `true`
# when using containerd with Kubernetes <=1.11.
disable_proc_mount = false
# unsetSeccompProfile is the profile containerd/cri will use if the provided seccomp profile is
# unset (`""`) for a container (default is `unconfined`)
unset_seccomp_profile = ""
# 'plugins."io.containerd.grpc.v1.cri".containerd' contains config related to containerd
[plugins."io.containerd.grpc.v1.cri".containerd]
# snapshotter is the snapshotter used by containerd.
snapshotter = "overlayfs"
# no_pivot disables pivot-root (linux only), required when running a container in a RamDisk with runc.
# This only works for runtime type "io.containerd.runtime.v1.linux".
no_pivot = false
# disable_snapshot_annotations disables to pass additional annotations (image
# related information) to snapshotters. These annotations are required by
# stargz snapshotter (https://github.com/containerd/stargz-snapshotter)
disable_snapshot_annotations = false
# discard_unpacked_layers allows GC to remove layers from the content store after
# successfully unpacking these layers to the snapshotter.
discard_unpacked_layers = false
# default_runtime_name is the default runtime name to use.
default_runtime_name = "runc"
# 'plugins."io.containerd.grpc.v1.cri".containerd.default_runtime' is the runtime to use in containerd.
# DEPRECATED: use `default_runtime_name` and `plugins."io.containerd.grpc.v1.cri".runtimes` instead.
# Remove in containerd 1.4.
[plugins."io.containerd.grpc.v1.cri".containerd.default_runtime]
# 'plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime' is a runtime to run untrusted workloads on it.
# DEPRECATED: use `untrusted` runtime in `plugins."io.containerd.grpc.v1.cri".runtimes` instead.
# Remove in containerd 1.4.
[plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime]
# 'plugins."io.containerd.grpc.v1.cri".containerd.runtimes' is a map from CRI RuntimeHandler strings, which specify types
# of runtime configurations, to the matching configurations.
# In this example, 'runc' is the RuntimeHandler string to match.
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
# runtime_type is the runtime type to use in containerd.
# The default value is "io.containerd.runc.v2" since containerd 1.4.
# The default value was "io.containerd.runc.v1" in containerd 1.3, "io.containerd.runtime.v1.linux" in prior releases.
runtime_type = "io.containerd.runc.v2"
# pod_annotations is a list of pod annotations passed to both pod
# sandbox as well as container OCI annotations. Pod_annotations also
# supports golang path match pattern - https://golang.org/pkg/path/#Match.
# e.g. ["runc.com.*"], ["*.runc.com"], ["runc.com/*"].
#
# For the naming convention of annotation keys, please reference:
# * Kubernetes: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/#syntax-and-character-set
# * OCI: https://github.com/opencontainers/image-spec/blob/master/annotations.md
pod_annotations = []
# container_annotations is a list of container annotations passed through to the OCI config of the containers.
# Container annotations in CRI are usually generated by other Kubernetes node components (i.e., not users).
# Currently, only device plugins populate the annotations.
container_annotations = []
# privileged_without_host_devices allows overloading the default behaviour of passing host
# devices through to privileged containers. This is useful when using a runtime where it does
# not make sense to pass host devices to the container when privileged. Defaults to false -
# i.e pass host devices through to privileged containers.
privileged_without_host_devices = false
# base_runtime_spec is a file path to a JSON file with the OCI spec that will be used as the base spec that all
# container's are created from.
# Use containerd's `ctr oci spec > /etc/containerd/cri-base.json` to output initial spec file.
# Spec files are loaded at launch, so containerd daemon must be restared on any changes to refresh default specs.
# Still running containers and restarted containers will still be using the original spec from which that container was created.
base_runtime_spec = ""
# 'plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options' is options specific to
# "io.containerd.runc.v1" and "io.containerd.runc.v2". Its corresponding options type is:
# https://github.com/containerd/containerd/blob/v1.3.2/runtime/v2/runc/options/oci.pb.go#L26 .
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
# NoPivotRoot disables pivot root when creating a container.
NoPivotRoot = false
# NoNewKeyring disables new keyring for the container.
NoNewKeyring = false
# ShimCgroup places the shim in a cgroup.
ShimCgroup = ""
# IoUid sets the I/O's pipes uid.
IoUid = 0
# IoGid sets the I/O's pipes gid.
IoGid = 0
# BinaryName is the binary name of the runc binary.
BinaryName = ""
# Root is the runc root directory.
Root = ""
# CriuPath is the criu binary path.
CriuPath = ""
# SystemdCgroup enables systemd cgroups.
SystemdCgroup = false
# CriuImagePath is the criu image path
CriuImagePath = ""
# CriuWorkPath is the criu work path.
CriuWorkPath = ""
# 'plugins."io.containerd.grpc.v1.cri".cni' contains config related to cni
[plugins."io.containerd.grpc.v1.cri".cni]
# bin_dir is the directory in which the binaries for the plugin is kept.
bin_dir = "/opt/cni/bin"
# conf_dir is the directory in which the admin places a CNI conf.
conf_dir = "/etc/cni/net.d"
# max_conf_num specifies the maximum number of CNI plugin config files to
# load from the CNI config directory. By default, only 1 CNI plugin config
# file will be loaded. If you want to load multiple CNI plugin config files
# set max_conf_num to the number desired. Setting max_config_num to 0 is
# interpreted as no limit is desired and will result in all CNI plugin
# config files being loaded from the CNI config directory.
max_conf_num = 1
# conf_template is the file path of golang template used to generate
# cni config.
# If this is set, containerd will generate a cni config file from the
# template. Otherwise, containerd will wait for the system admin or cni
# daemon to drop the config file into the conf_dir.
# This is a temporary backward-compatible solution for kubenet users
# who don't have a cni daemonset in production yet.
# This will be deprecated when kubenet is deprecated.
# See the "CNI Config Template" section for more details.
conf_template = ""
# 'plugins."io.containerd.grpc.v1.cri".registry' contains config related to the registry
[plugins."io.containerd.grpc.v1.cri".registry]
# 'plugins."io.containerd.grpc.v1.cri.registry.headers sets the http request headers to send for all registry requests
[plugins."io.containerd.grpc.v1.cri".registry.headers]
Foo = ["bar"]
# 'plugins."io.containerd.grpc.v1.cri".registry.mirrors' are namespace to mirror mapping for all namespaces.
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://registry-1.docker.io", ]
# 'plugins."io.containerd.grpc.v1.cri".image_decryption' contains config related
# to handling decryption of encrypted container images.
[plugins."io.containerd.grpc.v1.cri".image_decryption]
# key_model defines the name of the key model used for how the cri obtains
# keys used for decryption of encrypted container images.
# The [decryption document](https://github.com/containerd/cri/blob/master/docs/decryption.md)
# contains additional information about the key models available.
#
# Set of available string options: {"", "node"}
# Omission of this field defaults to the empty string "", which indicates no key model,
# disabling image decryption.
#
# In order to use the decryption feature, additional configurations must be made.
# The [decryption document](https://github.com/containerd/cri/blob/master/docs/decryption.md)
# provides information of how to set up stream processors and the containerd imgcrypt decoder
# with the appropriate key models.
#
# Additional information:
# * Stream processors: https://github.com/containerd/containerd/blob/master/docs/stream_processors.md
# * Containerd imgcrypt: https://github.com/containerd/imgcrypt
key_model = "node"
```
## Untrusted Workload
The recommended way to run untrusted workload is to use
[`RuntimeClass`](https://kubernetes.io/docs/concepts/containers/runtime-class/) api
introduced in Kubernetes 1.12 to select RuntimeHandlers configured to run
untrusted workload in `plugins."io.containerd.grpc.v1.cri".containerd.runtimes`.
However, if you are using the legacy `io.kubernetes.cri.untrusted-workload`pod annotation
to request a pod be run using a runtime for untrusted workloads, the RuntimeHandler
`plugins."io.containerd.grpc.v1.cri"cri.containerd.runtimes.untrusted` must be defined first.
When the annotation `io.kubernetes.cri.untrusted-workload` is set to `true` the `untrusted`
runtime will be used. For example, see
[Create an untrusted pod using Kata Containers](https://github.com/kata-containers/documentation/blob/master/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md#create-an-untrusted-pod-using-kata-containers).
## CNI Config Template
Ideally the cni config should be placed by system admin or cni daemon like calico,
weaveworks etc. However, there are still users using [kubenet](https://kubernetes.io/docs/concepts/cluster-administration/network-plugins/#kubenet)
today, who don't have a cni daemonset in production. The cni config template is
a temporary backward-compatible solution for them. This is expected to be
deprecated when kubenet is deprecated.
The cni config template uses the [golang
template](https://golang.org/pkg/text/template/) format. Currently supported
values are:
* `.PodCIDR` is a string of the first CIDR assigned to the node.
* `.PodCIDRRanges` is a string array of all CIDRs assigned to the node. It is
usually used for
[dualstack](https://github.com/kubernetes/enhancements/blob/master/keps/sig-network/20180612-ipv4-ipv6-dual-stack.md) support.
* `.Routes` is a string array of all routes needed. It is usually used for
dualstack support or single stack but IPv4 or IPv6 is decided at runtime.
The [golang template actions](https://golang.org/pkg/text/template/#hdr-Actions)
can be used to render the cni config. For example, you can use the following
template to add CIDRs and routes for dualstack in the CNI config:
```
"ipam": {
"type": "host-local",
"ranges": [{{range $i, $range := .PodCIDRRanges}}{{if $i}}, {{end}}[{"subnet": "{{$range}}"}]{{end}}],
"routes": [{{range $i, $route := .Routes}}{{if $i}}, {{end}}{"dst": "{{$route}}"}{{end}}]
}
```
## Deprecation
The config options of the CRI plugin follow the [Kubernetes deprecation
policy of "admin-facing CLI components"](https://kubernetes.io/docs/reference/using-api/deprecation-policy/#deprecating-a-flag-or-cli).
In summary, when a config option is announced to be deprecated:
* It is kept functional for 6 months or 1 release (whichever is longer);
* A warning is emitted when it is used.

BIN
docs/containerd.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 52 KiB

BIN
docs/cri.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 80 KiB

216
docs/crictl.md Normal file
View File

@ -0,0 +1,216 @@
CRICTL User Guide
=================
This document presumes you already have `containerd` with the `cri` plugin installed and running.
This document is for developers who wish to debug, inspect, and manage their pods,
containers, and container images.
Before generating issues against this document, `containerd`, `containerd/cri`,
or `crictl` please make sure the issue has not already been submitted.
## Install crictl
If you have not already installed crictl please install the version compatible
with the `cri` plugin you are using. If you are a user, your deployment
should have installed crictl for you. If not, get it from your release tarball.
If you are a developer the current version of crictl is specified [here](../hack/utils.sh).
A helper command has been included to install the dependencies at the right version:
```console
$ make install.deps
```
* Note: The file named `/etc/crictl.yaml` is used to configure crictl
so you don't have to repeatedly specify the runtime sock used to connect crictl
to the container runtime:
```console
$ cat /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: true
```
## Download and Inspect a Container Image
The pull command tells the container runtime to download a container image from
a container registry.
```console
$ crictl pull busybox
...
$ crictl inspecti busybox
... displays information about the image.
```
***Note:*** If you get an error similar to the following when running a `crictl`
command (and your containerd instance is already running):
```console
crictl info
FATA[0000] getting status of runtime failed: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
```
This could be that you are using an incorrect containerd configuration (maybe
from a Docker install). You will need to update your containerd configuration
to the containerd instance that you are running. One way of doing this is as
follows:
```console
$ mv /etc/containerd/config.toml /etc/containerd/config.bak
$ containerd config default > /etc/containerd/config.toml
```
## Directly Load a Container Image
Another way to load an image into the container runtime is with the load
command. With the load command you inject a container image into the container
runtime from a file. First you need to create a container image tarball. For
example to create an image tarball for a pause container using Docker:
```console
$ docker pull k8s.gcr.io/pause-amd64:3.2
3.2: Pulling from pause-amd64
67ddbfb20a22: Pull complete
Digest: sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610
Status: Downloaded newer image for k8s.gcr.io/pause-amd64:3.2
$ docker save k8s.gcr.io/pause-amd64:3.2 -o pause.tar
```
Then use [`ctr`](https://github.com/containerd/containerd/blob/master/docs/man/ctr.1.md)
to load the container image into the container runtime:
```console
# The cri plugin uses the "k8s.io" containerd namespace.
$ sudo ctr -n=k8s.io images import pause.tar
Loaded image: k8s.gcr.io/pause-amd64:3.2
```
List images and inspect the pause image:
```console
$ sudo crictl images
IMAGE TAG IMAGE ID SIZE
docker.io/library/busybox latest f6e427c148a76 728kB
k8s.gcr.io/pause-amd64 3.2 da86e6ba6ca19 746kB
$ sudo crictl inspecti da86e6ba6ca19
... displays information about the pause image.
$ sudo crictl inspecti k8s.gcr.io/pause-amd64:3.2
... displays information about the pause image.
```
## Run a pod sandbox (using a config file)
```console
$ cat sandbox-config.json
{
"metadata": {
"name": "nginx-sandbox",
"namespace": "default",
"attempt": 1,
"uid": "hdishd83djaidwnduwk28bcsb"
},
"linux": {
}
}
$ crictl runp sandbox-config.json
e1c83b0b8d481d4af8ba98d5f7812577fc175a37b10dc824335951f52addbb4e
$ crictl pods
PODSANDBOX ID CREATED STATE NAME NAMESPACE ATTEMPT
e1c83b0b8d481 2 hours ago SANDBOX_READY nginx-sandbox default 1
$ crictl inspectp e1c8
... displays information about the pod and the pod sandbox pause container.
```
* Note: As shown above, you may use truncated IDs if they are unique.
* Other commands to manage the pod include `stops ID` to stop a running pod and
`rmp ID` to remove a pod sandbox.
## Create and Run a Container in the Pod Sandbox (using a config file)
```console
$ cat container-config.json
{
"metadata": {
"name": "busybox"
},
"image":{
"image": "busybox"
},
"command": [
"top"
],
"linux": {
}
}
$ crictl create e1c83 container-config.json sandbox-config.json
0a2c761303163f2acaaeaee07d2ba143ee4cea7e3bde3d32190e2a36525c8a05
$ crictl ps -a
CONTAINER ID IMAGE CREATED STATE NAME ATTEMPT
0a2c761303163 docker.io/busybox 2 hours ago CONTAINER_CREATED busybox 0
$ crictl start 0a2c
0a2c761303163f2acaaeaee07d2ba143ee4cea7e3bde3d32190e2a36525c8a05
$ crictl ps
CONTAINER ID IMAGE CREATED STATE NAME ATTEMPT
0a2c761303163 docker.io/busybox 2 hours ago CONTAINER_RUNNING busybox 0
$ crictl inspect 0a2c7
... show detailed information about the container
```
## Exec a Command in the Container
```console
$ crictl exec -i -t 0a2c ls
bin dev etc home proc root sys tmp usr var
```
## Display Stats for the Container
```console
$ crictl stats
CONTAINER CPU % MEM DISK INODES
0a2c761303163f 0.00 983kB 16.38kB 6
```
* Other commands to manage the container include `stop ID` to stop a running
container and `rm ID` to remove a container.
## Display Version Information
```console
$ crictl version
Version: 0.1.0
RuntimeName: containerd
RuntimeVersion: 1.0.0-beta.1-186-gdd47a72-TEST
RuntimeApiVersion: v1alpha2
```
## Display Status & Configuration Information about Containerd & The CRI Plugin
```console
$ crictl info
{
"status": {
"conditions": [
{
"type": "RuntimeReady",
"status": true,
"reason": "",
"message": ""
},
{
"type": "NetworkReady",
"status": true,
"reason": "",
"message": ""
}
]
},
"config": {
"containerd": {
"snapshotter": "overlayfs",
"runtime": "io.containerd.runtime.v1.linux"
},
"cni": {
"binDir": "/opt/cni/bin",
"confDir": "/etc/cni/net.d"
},
"registry": {
"mirrors": {
"docker.io": {
"endpoint": [
"https://registry-1.docker.io"
]
}
}
},
"streamServerPort": "10010",
"sandboxImage": "k8s.gcr.io/pause:3.2",
"statsCollectPeriod": 10,
"containerdRootDir": "/var/lib/containerd",
"containerdEndpoint": "unix:///run/containerd/containerd.sock",
"rootDir": "/var/lib/containerd/io.containerd.grpc.v1.cri",
"stateDir": "/run/containerd/io.containerd.grpc.v1.cri",
},
"golang": "go1.10"
}
```
## More Information
See [here](https://github.com/kubernetes-sigs/cri-tools/blob/master/docs/crictl.md)
for information about crictl.

46
docs/decryption.md Normal file
View File

@ -0,0 +1,46 @@
# Configure Image Decryption
This document describes the method to configure encrypted container image decryption for `containerd` for use with the `cri` plugin.
## Encrypted Container Images
Encrypted container images are OCI images which contain encrypted blobs. These encrypted images can be created through the use of [containerd/imgcrypt project](https://github.com/containerd/imgcrypt). To decrypt these images, the `containerd` runtime uses information passed from the `cri` such as keys, options and encryption metadata.
## The "node" Key Model
Encryption ties trust to an entity based on the model in which a key is associated with it. We call this the key model. One such usecase is when we want to tie the trust of a key to the node in a cluster. In this case, we call it the "node" or "host" Key Model. Future work will include more key models to facilitate other trust associations (i.e. for multi-tenancy).
### "node" Key Model Usecase
In this model encryption is tied to worker nodes. The usecase here revolves around the idea that an image should be decryptable only on trusted host. Using this model, various node based technologies which help bootstrap trust in worker nodes and perform secure key distribution (i.e. TPM, host attestation, secure/measured boot). In this scenario, runtimes are capable of fetching the necessary decryption keys. An example of this is using the [`--decryption-keys-path` flag in imgcrypt](https://github.com/containerd/imgcrypt).
### Configuring image decryption for "node" key model
The default configuration does not handle decrypting encrypted container images.
An example for configuring the "node" key model for container image decryption:
Configure `cri` to enable decryption with "node" key model
```toml
[plugins."io.containerd.grpc.v1.cri".image_decryption]
key_model = "node"
```
Configure `containerd` daemon [`stream_processors`](https://github.com/containerd/containerd/blob/master/docs/stream_processors.md) to handle the
encrypted mediatypes.
```toml
[stream_processors]
[stream_processors."io.containerd.ocicrypt.decoder.v1.tar.gzip"]
accepts = ["application/vnd.oci.image.layer.v1.tar+gzip+encrypted"]
returns = "application/vnd.oci.image.layer.v1.tar+gzip"
path = "/usr/local/bin/ctd-decoder"
args = ["--decryption-keys-path", "/keys"]
[stream_processors."io.containerd.ocicrypt.decoder.v1.tar"]
accepts = ["application/vnd.oci.image.layer.v1.tar+encrypted"]
returns = "application/vnd.oci.image.layer.v1.tar"
path = "/usr/local/bin/ctd-decoder"
args = ["--decryption-keys-path", "/keys"]
```
In this example, container image decryption is set to use the "node" key model. In addition, the decryption [`stream_processors`](https://github.com/containerd/containerd/blob/master/docs/stream_processors.md) are configured as specified in [containerd/imgcrypt project](https://github.com/containerd/imgcrypt), with the additional field `--decryption-keys-path` configured to specify where decryption keys are located locally in the node.
After modify this config, you need restart the `containerd` service.

108
docs/installation.md Normal file
View File

@ -0,0 +1,108 @@
# Install Containerd with Release Tarball
This document provides the steps to install `containerd` and its dependencies with the release tarball, and bring up a Kubernetes cluster using kubeadm.
These steps have been verified on Ubuntu 16.04. For other OS distributions, the steps may differ. Please feel free to file issues or PRs if you encounter any problems on other OS distributions.
*Note: You need to run the following steps on each node you are planning to use in your Kubernetes cluster.*
## Release Tarball
For each `containerd` release, we'll publish a release tarball specifically for Kubernetes named `cri-containerd-${VERSION}.${OS}-${ARCH}.tar.gz`. This release tarball contains all required binaries and files for using `containerd` with Kubernetes. For example, the 1.2.4 version is available at https://storage.googleapis.com/cri-containerd-release/cri-containerd-1.2.4.linux-amd64.tar.gz.
Note: The VERSION tag specified for the tarball corresponds to the `containerd` release tag, not a containerd/cri repository release tag. The `containerd` release includes the containerd/cri repository code through vendoring. The containerd/cri version of the containerd/cri code included in `containerd` is specified via a commit hash for containerd/cri in containerd/containerd/vendor.conf.
### Content
As shown below, the release tarball contains:
1) `containerd`, `containerd-shim`, `containerd-stress`, `containerd-release`, `ctr`: binaries for containerd.
2) `runc`: runc binary.
3) `crictl`, `crictl.yaml`: command line tools for CRI container runtime and its config file.
4) `critest`: binary to run [CRI validation test](https://github.com/kubernetes-sigs/cri-tools/blob/master/docs/validation.md).
5) `containerd.service`: Systemd unit for containerd.
6) `/opt/containerd/cluster/`: scripts for `kube-up.sh`.
```console
$ tar -tf cri-containerd-1.1.0-rc.0.linux-amd64.tar.gz
./
./opt
./opt/containerd
./opt/containerd/cluster
./opt/containerd/cluster/gce
./opt/containerd/cluster/gce/cloud-init
./opt/containerd/cluster/gce/cloud-init/node.yaml
./opt/containerd/cluster/gce/cloud-init/master.yaml
./opt/containerd/cluster/gce/configure.sh
./opt/containerd/cluster/gce/env
./opt/containerd/cluster/version
./opt/containerd/cluster/health-monitor.sh
./usr
./usr/local
./usr/local/sbin
./usr/local/sbin/runc
./usr/local/bin
./usr/local/bin/crictl
./usr/local/bin/containerd
./usr/local/bin/containerd-stress
./usr/local/bin/critest
./usr/local/bin/containerd-release
./usr/local/bin/containerd-shim
./usr/local/bin/ctr
./etc
./etc/systemd
./etc/systemd/system
./etc/systemd/system/containerd.service
./etc/crictl.yaml
```
### Binary Information
Information about the binaries in the release tarball:
| Binary Name | Support | OS | Architecture |
|:------------------------------:|:------------------:|:-----:|:------------:|
| containerd | seccomp, apparmor, <br/> overlay, btrfs | linux | amd64 |
| containerd-shim | overlay, btrfs | linux | amd64 |
| runc | seccomp, apparmor | linux | amd64 |
If you have other requirements for the binaries, e.g. selinux support, another architecture support etc., you need to build the binaries yourself following [the instructions](../README.md#getting-started-for-developers).
### Download
The release tarball could be downloaded from the release GCS bucket https://storage.googleapis.com/cri-containerd-release/.
## Step 0: Install Dependent Libraries
Install required library for seccomp.
```bash
sudo apt-get update
sudo apt-get install libseccomp2
```
Note that:
1) If you are using Ubuntu <=Trusty or Debian <=jessie, a backported version of `libseccomp2` is needed. (See the [trusty-backports](https://packages.ubuntu.com/trusty-backports/libseccomp2) and [jessie-backports](https://packages.debian.org/jessie-backports/libseccomp2)).
## Step 1: Download Release Tarball
Download release tarball for the `containerd` version you want to install from the GCS bucket.
```bash
wget https://storage.googleapis.com/cri-containerd-release/cri-containerd-${VERSION}.linux-amd64.tar.gz
```
Validate checksum of the release tarball:
```bash
sha256sum cri-containerd-${VERSION}.linux-amd64.tar.gz
curl https://storage.googleapis.com/cri-containerd-release/cri-containerd-${VERSION}.linux-amd64.tar.gz.sha256
# Compare to make sure the 2 checksums are the same.
```
## Step 2: Install Containerd
If you are using systemd, just simply unpack the tarball to the root directory:
```bash
sudo tar --no-overwrite-dir -C / -xzf cri-containerd-${VERSION}.linux-amd64.tar.gz
sudo systemctl start containerd
```
If you are not using systemd, please unpack all binaries into a directory in your `PATH`, and start `containerd` as monitored long running services with the service manager you are using e.g. `supervisord`, `upstart` etc.
## Step 3: Install Kubeadm, Kubelet and Kubectl
Follow [the instructions](https://kubernetes.io/docs/setup/independent/install-kubeadm/) to install kubeadm, kubelet and kubectl.
## Step 4: Create Systemd Drop-In for Containerd
Create the systemd drop-in file `/etc/systemd/system/kubelet.service.d/0-containerd.conf`:
```
[Service]
Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock"
```
And reload systemd configuration:
```bash
systemctl daemon-reload
```
## Bring Up the Cluster
Now you should have properly installed all required binaries and dependencies on each of your node.
The next step is to use kubeadm to bring up the Kubernetes cluster. It is the same with [the ansible installer](../contrib/ansible). Please follow the steps 2-4 [here](../contrib/ansible/README.md#step-2).

25
docs/kube-up.md Normal file
View File

@ -0,0 +1,25 @@
# Production Quality Cluster on GCE
This document provides the steps to bring up a production quality cluster on GCE with [`kube-up.sh`](https://kubernetes.io/docs/setup/turnkey/gce/).
**If your Kubernetes version is 1.15 or greater, you can simply run:**
```
export KUBE_CONTAINER_RUNTIME=containerd
```
Follow these instructions [here](https://kubernetes.io/docs/setup/turnkey/gce/) to create a production quality Kubernetes cluster on GCE.
## Download CRI-Containerd Release Tarball
To download release tarball, see [step 1](./installation.md#step-1-download-cri-containerd-release-tarball) in installation.md.
Unpack release tarball to any directory, using `${CRI_CONTAINERD_PATH}` to indicate the directory in the doc:
```bash
tar -C ${CRI_CONTAINERD_PATH} -xzf cri-containerd-${VERSION}.linux-amd64.tar.gz
```
## Set Environment Variables for CRI-Containerd
```bash
. ${CRI_CONTAINERD_PATH}/opt/containerd/cluster/gce/env
```
## Create Kubernetes Cluster on GCE
Follow these instructions [here](https://kubernetes.io/docs/setup/turnkey/gce/) to create a production quality Kubernetes cluster on GCE.
**Make sure the Kubernetes version you are using is v1.11 or greater:**
* When using `https://get.k8s.io`, use the environment variable `KUBERNETES_RELEASE` to set version.
* When using a Kubernetes release tarball, make sure to select version 1.11 or greater.

BIN
docs/performance.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 69 KiB

111
docs/proposal.md Normal file
View File

@ -0,0 +1,111 @@
Containerd CRI Integration
=============
Author: Lantao Liu (@random-liu)
## Abstract
This proposal aims to integrate [containerd](https://github.com/containerd/containerd) with Kubelet against the [container runtime interface (CRI)](https://github.com/kubernetes/kubernetes/blob/v1.6.0/pkg/kubelet/api/v1alpha1/runtime/api.proto).
## Background
Containerd is a core container runtime, which provides the minimum set of functionalities to manage the complete container lifecycle of its host system, including container execution and supervision, image distribution and storage, etc.
Containerd was [introduced in Docker 1.11](https://blog.docker.com/2016/04/docker-engine-1-11-runc/), used to manage [runC](https://runc.io/) containers on the node. As shown below, it creates a containerd-shim for each container, and the shim manages the lifecycle of its corresponding container.
![containerd](./containerd.png)
In Dec. 2016, Docker Inc. spun it out into a standalone component, and donated it to [CNCF](https://www.cncf.io/) in Mar. 2017.
## Motivation
Containerd is one potential alternative to Docker as the runtime for Kubernetes clusters. *Compared with Docker*, containerd has pros and cons.
### Pros
* **Stability**: Containerd has limited scope and slower feature velocity, which is expected to be more stable.
* **Compatibility**: The scope of containerd aligns with Kubernetes' requirements. It provides the required functionalities and the flexibility for areas like image pulling, networking, volume and logging etc.
* **Performance**:
* Containerd consumes less resource than Docker at least because it's a subset of Docker;
* Containerd CRI integration eliminates an extra hop in the stack (as shown below). ![performance](./performance.png)
* **Neutral Foundation**: Containerd is part of CNCF now.
### Cons
* **User Adoption**:
* Ideally, Kubernetes users don't interact with the underlying container runtime directly. However, for the lack of debug toolkits, sometimes users still need to login the node to debug with Docker CLI directly.
* Containerd provides barebone CLIs [ctr](https://github.com/containerd/containerd/tree/master/cmd/ctr) and [dist](https://github.com/containerd/containerd/tree/master/cmd/dist) for development and debugging purpose, but they may not be sufficient and necessary. Additionally, presuming these are sufficient and necessary tools, a plan and time would be needed to sufficiently document these CLIs and educate users in their use.
* **Maturity**: The rescoped containerd is pretty new, and it's still under heavy development.
## Goals
* Make sure containerd meets the requirement of Kubernetes, now and into the foreseeable future.
* Implement containerd CRI shim and make sure it provides equivalent functionality, usability and debuggability.
* Improve Kubernetes by taking advantage of the flexibility provided by containerd.
## Design
The following sections discuss the design aspects of the containerd CRI integration. For the purposes of this doc, the containerd CRI integration will be referred to as `CRI-containerd`.
### Container Lifecycle
CRI-containerd relies on containerd to manage container lifecycle.
Ideally, CRI-containerd only needs to do api translation and information reorganization. However, CRI-containerd needs to maintain some metadata because:
* There is a mismatch between container lifecycle of CRI and containerd - containerd only tracks running processes, once the container and it's corresponding containerd-shim exit, the container is no longer visible in the containerd API.
* Some sandbox/container metadata is not provided by containerd, and we can not leverage OCI runtime annotation to store it because of the container lifecycle mismatch, e.g. labels/annotations, `PodSandboxID` of a container, `FinishedAt` timestamp, `ExitCode`, `Mounts` etc.
CRI-containerd should checkpoint these metadata itself or use [containerd metadata service](https://github.com/containerd/containerd/blob/0a5544d8c4dab44dfc682f5ad07f1cd011c0a115/design/plugins.md#core) if available.
### Container Logging
Containerd doesn't provide persistent container log. It redirects container STDIO into different FIFOs.
CRI-containerd should start a goroutine (process/container in the future) to:
* Continuously drain the FIFO;
* Decorate the log line into [CRI-defined format](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/kubelet-cri-logging.md#proposed-solution);
* Write the log into [CRI-defined log path](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/kubelet-cri-logging.md#proposed-solution).
### Container Streaming
Containerd supports creating a process in the container with `Exec`, and the STDIO is also exposed as FIFOs. Containerd also supports resizing console of a specific process with `Pty`.
CRI-containerd could reuse the [streaming server](https://github.com/kubernetes/kubernetes/blob/release-1.6/pkg/kubelet/server/streaming/server.go), it should implement the [streaming runtime interface](https://github.com/kubernetes/kubernetes/blob/release-1.6/pkg/kubelet/server/streaming/server.go#L61-L65).
For different CRI streaming functions:
* `ExecSync`: CRI-containerd should use `Exec` to create the exec process, collect the stdout/stderr of the process, and wait for the process to terminate.
* `Exec`: CRI-containerd should use `Exec` to create the exec process, create a goroutine (process/container) to redirect streams, and wait for the process to terminate.
* `Attach`: CRI-containerd should create a goroutine (process/container) to read the existing container log to the output, redirect streams of the init process, and wait for any stream to be closed.
* `PortForward`: CRI-containerd could implement this with `socat` and `nsenter`, similar with [current Docker portforward implementation](https://github.com/kubernetes/kubernetes/blob/release-1.6/pkg/kubelet/dockertools/docker_manager.go#L1373-L1428).
### Container Networking
Containerd doesn't provide container networking, but OCI runtime spec supports joining a linux container into an existing network namespace.
CRI-containerd should:
* Create a network namespace for a sandbox;
* Call [network plugin](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/network/plugins.go) to update the options of the network namespace;
* Let the user containers in the same sandbox share the network namespace.
### Container Metrics
Containerd provides [container cgroup metrics](https://github.com/containerd/containerd/blob/master/reports/2017-03-17.md#metrics), and plans to provide [container writable layer disk usage](https://github.com/containerd/containerd/issues/678).
CRI container metrics api needs to be defined ([#27097](https://github.com/kubernetes/kubernetes/issues/27097)). After that, CRI-containerd should translate containerd container metrics into CRI container metrics.
### Image Management
CRI-containerd relies on containerd to manage images. Containerd should provide all function and information required by CRI, and CRI-containerd only needs to do api translation and information reorganization.
### ImageFS Metrics
Containerd plans to provide [image filesystem metrics](https://github.com/containerd/containerd/issues/678).
CRI image filesystem metrics needs to be defined ([#33048](https://github.com/kubernetes/kubernetes/issues/33048)). After that, we should make sure containerd provides the required metrics, and CRI-containerd should translate containerd image filesystem metrics into CRI image filesystem metrics.
### Out of Scope
Following items are out of the scope of this design, we may address them in future version as enhancement or optimization.
* **Debuggability**: One of the biggest concern of CRI-containerd is debuggability. We should provide equivalent debuggability with Docker CLI through `kubectl`, [`cri-tools`](https://github.com/kubernetes-sigs/cri-tools) or containerd CLI.
* **Built-in CRI support**: The [plugin model](https://github.com/containerd/containerd/blob/master/design/plugins.md) provided by containerd makes it possible to directly build CRI support into containerd as a plugin, which will eliminate one more hop from the stack. But because of the [limitation of golang plugin](https://github.com/containerd/containerd/issues/563), we have to either maintain our own branch or push CRI plugin upstream.
* **Seccomp**: ([#36997](https://github.com/kubernetes/kubernetes/issues/36997)) Seccomp is supported in OCI runtime spec. However, current seccomp implementation in Kubernetes is experimental and docker specific, the api needs to be defined in CRI first before CRI-containerd implements it.
* **Streaming server authentication**: ([#36666](https://github.com/kubernetes/kubernetes/issues/36666)) CRI-containerd will be out-of-process with Kubelet, so it could not reuse Kubelet authentication. Its streaming server should implement its own authentication mechanism.
* **Move container facilities into pod cgroup**: Container facilities including container image puller, container streaming handler, log handler and containerd-shim serve a specific container. They should be moved to the corresponding pod cgroup, and the overhead introduced by them should be charged to the pod.
* **Log rotation**: ([#42718](https://github.com/kubernetes/kubernetes/issues/42718)) Container log rotation is under design. A function may be added in CRI to signal the runtime to reopen log file. CRI-containerd should implement that function after it is defined.
* **Exec container**: With the flexibility provided by containerd, it is possible to implement `Exec` with a separate container sharing the same rootfs and mount namespace with the original container. The advantage is that the `Exec` container could have it's own sub-cgroup, so that it will not consume the resource of application container and user could specify dedicated resource for it.
* **Advanced image management**: The image management interface in CRI is relatively simple because the requirement of Kubelet image management is not clearly scoped out. In the future, we may want to leverage the flexibility provided by containerd more, e.g. estimate image size before pulling etc.
* ...
## Roadmap and Milestones
### Milestones
#### Kubernetes 1.7 - Q2
* [P0] Basic container lifecycle.
* [P0] Basic image management.
* [P0] Container networking.
* [P1] Container streaming/logging.
* [P2] Container/ImageFS Metrics.
*Test Plan: Each feature added should have unit test and pass its corresponding cri validation test.*
#### Kubernetes 1.8 - Q3
* [P0] Feature complete, pass 100% cri validation test.
* [P0] Integrate CRI-containerd with Kubernetes, and build the e2e/node e2e test framework.
* [P1] Address the debuggability problem.
### Q2 Roadmap
| Item | 1/2 Mar. | 2/2 Mar. | 1/2 Apr. | 2/2 Apr. | 1/2 May. | 2/2 May. |
|:--------------------------------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|
| Survey | ✓ | | | | | |
| POC | | ✓ | | | | |
| Proposal | | | ✓ | | | |
| Containerd Feature Complete | ✓ | ✓ | ✓ | | | |
| Runtime Management Integration | | | ✓ | ✓ | ✓ | ✓ |
| Image Management Integration | | | | ✓ | ✓ | ✓ |
| Container Networking Integration | | | | | ✓ | ✓ |

187
docs/registry.md Normal file
View File

@ -0,0 +1,187 @@
# Configure Image Registry
This document describes the method to configure the image registry for `containerd` for use with the `cri` plugin.
NOTE: The configuration syntax used in this doc is in version 2 which is the
recommended since `containerd` 1.3. If your configuration is still in version 1,
you can replace `"io.containerd.grpc.v1.cri"` with `cri`.
## Configure Registry Endpoint
With containerd, `docker.io` is the default image registry. You can also set up other image registries similar to docker.
To configure image registries create/modify the `/etc/containerd/config.toml` as follows:
```toml
# Config file is parsed as version 1 by default.
# To use the long form of plugin names set "version = 2"
# explicitly use v2 config format
version = 2
[plugin."io.containerd.grpc.v1.cri".registry.mirrors]
[plugin."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://registry-1.docker.io"]
[plugin."io.containerd.grpc.v1.cri".registry.mirrors."test.https-registry.io"]
endpoint = ["https://HostIP1:Port1"]
[plugin."io.containerd.grpc.v1.cri".registry.mirrors."test.http-registry.io"]
endpoint = ["http://HostIP2:Port2"]
# wildcard matching is supported but not required.
[plugin."io.containerd.grpc.v1.cri".registry.mirrors."*"]
endpoint = ["https://HostIP3:Port3"]
```
The default configuration can be generated by `containerd config default > /etc/containerd/config.toml`.
The endpoint is a list that can contain multiple image registry URLs split by commas. When pulling an image
from a registry, containerd will try these endpoint URLs one by one, and use the first working one. Please note
that if the default registry endpoint is not already specified in the endpoint list, it will be automatically
tried at the end with scheme `https` and path `v2`, e.g. `https://gcr.io/v2` for `gcr.io`.
As an example, for the image `gcr.io/library/busybox:latest`, the endpoints are:
* `gcr.io` is configured: endpoints for `gcr.io` + default endpoint `https://gcr.io/v2`.
* `*` is configured, and `gcr.io` is not: endpoints for `*` + default
endpoint `https://gcr.io/v2`.
* None of above is configured: default endpoint `https://gcr.io/v2`.
After modify this config, you need restart the `containerd` service.
## Configure Registry TLS Communication
`cri` plugin also supports configuring TLS settings when communicating with a registry.
To configure the TLS settings for a specific registry, create/modify the `/etc/containerd/config.toml` as follows:
```toml
# explicitly use v2 config format
version = 2
# The registry host has to be a domain name or IP. Port number is also
# needed if the default HTTPS or HTTP port is not used.
[plugin."io.containerd.grpc.v1.cri".registry.configs."my.custom.registry".tls]
ca_file = "ca.pem"
cert_file = "cert.pem"
key_file = "key.pem"
```
In the config example shown above, TLS mutual authentication will be used for communications with the registry endpoint located at <https://my.custom.registry>.
`ca_file` is file name of the certificate authority (CA) certificate used to authenticate the x509 certificate/key pair specified by the files respectively pointed to by `cert_file` and `key_file`.
`cert_file` and `key_file` are not needed when TLS mutual authentication is unused.
```toml
# explicitly use v2 config format
version = 2
[plugin."io.containerd.grpc.v1.cri".registry.configs."my.custom.registry".tls]
ca_file = "ca.pem"
```
To skip the registry certificate verification:
```toml
# explicitly use v2 config format
version = 2
[plugin."io.containerd.grpc.v1.cri".registry.configs."my.custom.registry".tls]
insecure_skip_verify = true
```
## Configure Registry Credentials
`cri` plugin also supports docker like registry credential config.
To configure a credential for a specific registry, create/modify the
`/etc/containerd/config.toml` as follows:
```toml
# explicitly use v2 config format
version = 2
# The registry host has to be a domain name or IP. Port number is also
# needed if the default HTTPS or HTTP port is not used.
[plugin."io.containerd.grpc.v1.cri".registry.configs."gcr.io".auth]
username = ""
password = ""
auth = ""
identitytoken = ""
```
The meaning of each field is the same with the corresponding field in `.docker/config.json`.
Please note that auth config passed by CRI takes precedence over this config.
The registry credential in this config will only be used when auth config is
not specified by Kubernetes via CRI.
After modifying this config, you need to restart the `containerd` service.
### Configure Registry Credentials Example - GCR with Service Account Key Authentication
If you don't already have Google Container Registry (GCR) set-up then you need to do the following steps:
* Create a Google Cloud Platform (GCP) account and project if not already created (see [GCP getting started](https://cloud.google.com/gcp/getting-started))
* Enable GCR for your project (see [Quickstart for Container Registry](https://cloud.google.com/container-registry/docs/quickstart))
* For authentication to GCR: Create [service account and JSON key](https://cloud.google.com/container-registry/docs/advanced-authentication#json-key)
* The JSON key file needs to be downloaded to your system from the GCP console
* For access to the GCR storage: Add service account to the GCR storage bucket with storage admin access rights (see [Granting permissions](https://cloud.google.com/container-registry/docs/access-control#grant-bucket))
Refer to [Pushing and pulling images](https://cloud.google.com/container-registry/docs/pushing-and-pulling) for detailed information on the above steps.
> Note: The JSON key file is a multi-line file and it can be cumbersome to use the contents as a key outside of the file. It is worthwhile generating a single line format output of the file. One way of doing this is using the `jq` tool as follows: `jq -c . key.json`
It is beneficial to first confirm that from your terminal you can authenticate with your GCR and have access to the storage before hooking it into containerd. This can be verified by performing a login to your GCR and
pushing an image to it as follows:
```console
docker login -u _json_key -p "$(cat key.json)" gcr.io
docker pull busybox
docker tag busybox gcr.io/your-gcp-project-id/busybox
docker push gcr.io/your-gcp-project-id/busybox
docker logout gcr.io
```
Now that you know you can access your GCR from your terminal, it is now time to try out containerd.
Edit the containerd config (default location is at `/etc/containerd/config.toml`)
to add your JSON key for `gcr.io` domain image pull
requests:
```toml
version = 2
[plugins."io.containerd.grpc.v1.cri".registry]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://registry-1.docker.io"]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."gcr.io"]
endpoint = ["https://gcr.io"]
[plugins."io.containerd.grpc.v1.cri".registry.configs]
[plugins."io.containerd.grpc.v1.cri".registry.configs."gcr.io".auth]
username = "_json_key"
password = 'paste output from jq'
```
> Note: `username` of `_json_key` signifies that JSON key authentication will be used.
Restart containerd:
```console
service containerd restart
```
Pull an image from your GCR with `crictl`:
```console
$ sudo crictl pull gcr.io/your-gcp-project-id/busybox
DEBU[0000] get image connection
DEBU[0000] connect using endpoint 'unix:///run/containerd/containerd.sock' with '3s' timeout
DEBU[0000] connected successfully using endpoint: unix:///run/containerd/containerd.sock
DEBU[0000] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:gcr.io/your-gcr-instance-id/busybox,},Auth:nil,SandboxConfig:nil,}
DEBU[0001] PullImageResponse: &PullImageResponse{ImageRef:sha256:78096d0a54788961ca68393e5f8038704b97d8af374249dc5c8faec1b8045e42,}
Image is up to date for sha256:78096d0a54788961ca68393e5f8038704b97d8af374249dc5c8faec1b8045e42
```

27
docs/release.md Normal file
View File

@ -0,0 +1,27 @@
# Release Process
This document describes how to cut a `cri` plugin release.
## Step 1: Update containerd vendor
Update the version of containerd located in `containerd/cri/vendor.conf`
to the latest version of containerd for the desired branch of containerd,
and make sure all tests in CI pass https://k8s-testgrid.appspot.com/sig-node-containerd.
## Step 2: Cut the release
Draft and tag a new release in https://github.com/containerd/cri/releases.
## Step 3: Update `cri` version in containerd
Push a PR to `containerd/containerd` that updates the version of
`containerd/cri` in `containerd/containerd/vendor.conf` to the newly
tagged release created in Step 2.
## Step 4: Iterate step 1 updating containerd vendor
## Step 5: Publish release tarball for Kubernetes
Publish the release tarball `cri-containerd-${CONTAINERD_VERSION}.${OS}-${ARCH}.tar.gz`
```shell
# Checkout `containerd/cri` to the newly released version.
git checkout ${RELEASE_VERSION}
# Publish the release tarball without cni.
DEPLOY_BUCKET=cri-containerd-release make push TARBALL_PREFIX=cri-containerd OFFICIAL_RELEASE=true VERSION=${CONTAINERD_VERSION}
# Publish the release tarball with cni.
DEPLOY_BUCKET=cri-containerd-release make push TARBALL_PREFIX=cri-containerd-cni OFFICIAL_RELEASE=true INCLUDE_CNI=true VERSION=${CONTAINERD_VERSION}
```
## Step 6: Update release note with release tarball information

58
docs/testing.md Normal file
View File

@ -0,0 +1,58 @@
CRI Plugin Testing Guide
========================
This document assumes you have already setup the development environment (go, git, `containerd/cri` repo etc.).
Before sending pull requests you should at least make sure your changes have passed code verification, unit, integration and CRI validation tests.
## Code Verification
Code verification includes lint, and code formatting check etc.
* Install tools used by code verification:
```bash
make install.tools
```
***Note:*** Some make actions (like `install.tools`) use the user's `GOPATH` and will otherwise not work when it is not set. Other make actions override it by setting it to a temporary directory for release build and testing purposes.
* Run code verification:
```bash
make verify
```
## Unit Test
Run all unit tests in `containerd/cri` repo.
```bash
make test
```
## Integration Test
Run all integration tests in `containerd/cri` repo.
* [Install dependencies](../README.md#install-dependencies).
* Run integration test:
```bash
make test-integration
```
## CRI Validation Test
[CRI validation test](https://github.com/kubernetes/community/blob/master/contributors/devel/cri-validation.md) is a test framework for validating that a Container Runtime Interface (CRI) implementation such as containerd with the `cri` plugin meets all the requirements necessary to manage pod sandboxes, containers, images etc.
CRI validation test makes it possible to verify CRI conformance of `containerd/cri` without setting up Kubernetes components or running Kubernetes end-to-end tests.
* [Install dependencies](../README.md#install-dependencies).
* Build containerd with the `cri` plugin:
```bash
make
```
* Run CRI validation test:
```bash
make test-cri
```
* Focus or skip specific CRI validation test:
```bash
make test-cri FOCUS=REGEXP_TO_FOCUS SKIP=REGEXP_TO_SKIP
```
[More information](https://github.com/kubernetes-sigs/cri-tools) about CRI validation test.
## Node E2E Test
[Node e2e test](https://github.com/kubernetes/community/blob/master/contributors/devel/e2e-node-tests.md) is a test framework testing Kubernetes node level functionalities such as managing pods, mounting volumes etc. It starts a local cluster with Kubelet and a few other minimum dependencies, and runs node functionality tests against the local cluster.
* [Install dependencies](../README.md#install-dependencies).
* Run node e2e test:
```bash
make test-e2e-node
```
* Focus or skip specific node e2e test:
```bash
make test-e2e-node FOCUS=REGEXP_TO_FOCUS SKIP=REGEXP_TO_SKIP
```
[More information](https://github.com/kubernetes/community/blob/master/contributors/devel/e2e-node-tests.md) about Kubernetes node e2e test.

15
hack/boilerplate/boilerplate Normal file
View File

@ -0,0 +1,15 @@
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

57
hack/install/install-cni-config.sh Executable file
View File

@ -0,0 +1,57 @@
#!/bin/bash
# Copyright The containerd Authors.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
set -o errexit
set -o nounset
set -o pipefail
source $(dirname "${BASH_SOURCE[0]}")/utils.sh
CNI_CONFIG_DIR=${DESTDIR}/etc/cni/net.d
${SUDO} mkdir -p ${CNI_CONFIG_DIR}
${SUDO} bash -c 'cat >'${CNI_CONFIG_DIR}'/10-containerd-net.conflist <<EOF
{
"cniVersion": "0.4.0",
"name": "containerd-net",
"plugins": [
{
"type": "bridge",
"bridge": "cni0",
"isGateway": true,
"ipMasq": true,
"promiscMode": true,
"ipam": {
"type": "host-local",
"ranges": [
[{
"subnet": "10.88.0.0/16"
}],
[{
"subnet": "2001:4860:4860::/64"
}]
],
"routes": [
{ "dst": "0.0.0.0/0" },
{ "dst": "::/0" }
]
}
},
{
"type": "portmap",
"capabilities": {"portMappings": true}
}
]
}
EOF'

45
hack/install/install-cni.sh Executable file
View File

@ -0,0 +1,45 @@
#!/bin/bash
# Copyright The containerd Authors.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
set -o errexit
set -o nounset
set -o pipefail
source $(dirname "${BASH_SOURCE[0]}")/utils.sh
CNI_DIR=${DESTDIR}/opt/cni
CNI_PKG=github.com/containernetworking/plugins
# Create a temporary GOPATH for cni installation.
GOPATH=$(mktemp -d /tmp/cri-install-cni.XXXX)
# Install cni
from-vendor CNI github.com/containernetworking/plugins
checkout_repo ${CNI_PKG} ${CNI_VERSION} ${CNI_REPO}
cd ${GOPATH}/src/${CNI_PKG}
if [[ "$OSTYPE" == "linux-gnu"* ]] || [[ "$OSTYPE" == "darwin"* ]]; then
./build_linux.sh
elif [[ "$OSTYPE" == "win32" ]]; then
./build_windows.sh
else
exit 1
fi
${SUDO} mkdir -p ${CNI_DIR}
${SUDO} cp -r ./bin ${CNI_DIR}
# Clean the tmp GOPATH dir.
rm -rf ${GOPATH}

49
hack/install/install-containerd.sh Executable file
View File

@ -0,0 +1,49 @@
#!/bin/bash
# Copyright The containerd Authors.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
set -o errexit
set -o nounset
set -o pipefail
source $(dirname "${BASH_SOURCE[0]}")/utils.sh
CONTAINERD_DIR=${CONTAINERD_DIR:-"${DESTDIR}/usr/local"}
CONTAINERD_PKG=github.com/containerd/containerd
# CHECKOUT_CONTAINERD indicates whether to checkout containerd repo.
# This is useful for build containerd from existing repo, currently
# used by containerd CI test.
CHECKOUT_CONTAINERD=${CHECKOUT_CONTAINERD:-true}
if ${CHECKOUT_CONTAINERD}; then
# Create a temporary GOPATH for containerd installation.
export GOPATH=$(mktemp -d /tmp/cri-install-containerd.XXXX)
from-vendor CONTAINERD github.com/containerd/containerd
checkout_repo ${CONTAINERD_PKG} ${CONTAINERD_VERSION} ${CONTAINERD_REPO}
fi
# Install containerd
cd ${GOPATH}/src/${CONTAINERD_PKG}
make BUILDTAGS="${BUILDTAGS}"
# containerd make install requires `go` to work. Explicitly
# set PATH to make sure it can find `go` even with `sudo`.
# The single quote is required because containerd Makefile
# can't handle spaces in the path.
${SUDO} make install -e DESTDIR="'${CONTAINERD_DIR}'"
# Clean the tmp GOPATH dir.
if ${CHECKOUT_CONTAINERD}; then
rm -rf ${GOPATH}
fi

40
hack/install/install-critools.sh Executable file
View File

@ -0,0 +1,40 @@
#!/bin/bash
# Copyright The containerd Authors.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
set -o errexit
set -o nounset
set -o pipefail
source $(dirname "${BASH_SOURCE[0]}")/utils.sh
CRITOOL_DIR="${CRITOOL_DIR:-${DESTDIR}/usr/local/bin}"
CRICTL_CONFIG_DIR="${CRICTL_CONFIG_DIR:-"${DESTDIR}/etc"}"
CRICTL_RUNTIME_ENDPOINT=${CRICTL_RUNTIME_ENDPOINT:-unix:///run/containerd/containerd.sock}
# Create a temporary GOPATH for crictl installation.
GOPATH=$(mktemp -d /tmp/cri-install-crictl.XXXX)
#Install crictl
checkout_repo ${CRITOOL_PKG} ${CRITOOL_VERSION} ${CRITOOL_REPO}
cd ${GOPATH}/src/${CRITOOL_PKG}
make VERSION=${CRITOOL_VERSION}
${SUDO} make install -e BINDIR="\"${CRITOOL_DIR}\"" GOPATH=${GOPATH}
${SUDO} mkdir -p ${CRICTL_CONFIG_DIR}
${SUDO} bash -c 'cat >"'"${CRICTL_CONFIG_DIR}"'"/crictl.yaml <<EOF
runtime-endpoint: '${CRICTL_RUNTIME_ENDPOINT}'
EOF'
# Clean the tmp GOPATH dir.
rm -rf ${GOPATH}

44
hack/install/install-deps.sh Executable file
View File

@ -0,0 +1,44 @@
#!/bin/bash
# Copyright The containerd Authors.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Dependencies:
# runc:
# - libseccomp-dev(Ubuntu,Debian)/libseccomp-devel(Fedora, CentOS, RHEL). Note that
# libseccomp in ubuntu <=trusty and debian <=jessie is not new enough, backport
# is required.
# containerd:
# - btrfs-tools(Ubuntu,Debian)/btrfs-progs-devel(Fedora, CentOS, RHEL)
set -o errexit
set -o nounset
set -o pipefail
cd $(dirname "${BASH_SOURCE[0]}")
# Install runc
./install-runc.sh
# Install cni
./install-cni.sh
# Install cni config
./install-cni-config.sh
# Install containerd
./install-containerd.sh
#Install critools
./install-critools.sh

37
hack/install/install-runc.sh Executable file
View File

@ -0,0 +1,37 @@
#!/bin/bash
# Copyright The containerd Authors.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
set -o errexit
set -o nounset
set -o pipefail
source $(dirname "${BASH_SOURCE[0]}")/utils.sh
RUNC_DIR=${DESTDIR}
RUNC_PKG=github.com/opencontainers/runc
# Create a temporary GOPATH for runc installation.
GOPATH=$(mktemp -d /tmp/cri-install-runc.XXXX)
# Install runc
from-vendor RUNC github.com/opencontainers/runc
checkout_repo ${RUNC_PKG} ${RUNC_VERSION} ${RUNC_REPO}
cd ${GOPATH}/src/${RUNC_PKG}
make BUILDTAGS="$BUILDTAGS" VERSION=${RUNC_VERSION}
${SUDO} make install -e DESTDIR=${RUNC_DIR}
# Clean the tmp GOPATH dir. Use sudo because runc build generates
# some privileged files.
${SUDO} rm -rf ${GOPATH}

55
hack/install/utils.sh Executable file
View File

@ -0,0 +1,55 @@
#!/bin/bash
# Copyright The containerd Authors.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
source $(dirname "${BASH_SOURCE[0]}")/../utils.sh
# DESTDIR is the absolute dest path to install dependencies.
DESTDIR=${DESTDIR:-"/"}
# Make sure that DESTDIR is an absolute path.
if [[ ${DESTDIR} != /* ]]; then
echo "DESTDIR is not an absolute path"
exit 1
fi
# NOSUDO indicates not to use sudo during installation.
NOSUDO=${NOSUDO:-false}
SUDO="sudo"
if ${NOSUDO}; then
SUDO=""
fi
# BUILDTAGS are bulid tags for runc and containerd.
BUILDTAGS=${BUILDTAGS:-seccomp apparmor selinux}
# checkout_repo checks out specified repository
# and switch to specified version.
# Varset:
# 1) Pkg name;
# 2) Version;
# 3) Repo name;
checkout_repo() {
local -r pkg=$1
local -r version=$2
local -r repo=$3
path="${GOPATH}/src/${pkg}"
if [ ! -d ${path} ]; then
mkdir -p ${path}
git clone https://${repo} ${path}
fi
cd ${path}
git fetch --all
git checkout ${version}
}

86
hack/install/windows/install-cni-config.sh Executable file
View File

@ -0,0 +1,86 @@
#!/bin/bash
# Copyright The containerd Authors.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
set -o errexit
set -o nounset
set -o pipefail
CNI_CONFIG_DIR="${CNI_CONFIG_DIR:-"C:\\Program Files\\containerd\\cni\\conf"}"
mkdir -p "${CNI_CONFIG_DIR}"
# split_ip splits ip into a 4-element array.
split_ip() {
local -r varname="$1"
local -r ip="$2"
for i in {0..3}; do
eval "$varname"[$i]=$( echo "$ip" | cut -d '.' -f $((i + 1)) )
done
}
# subnet gets subnet for a gateway, e.g. 192.168.100.0/24.
calculate_subnet() {
local -r gateway="$1"
local -r prefix_len="$2"
split_ip gateway_array "$gateway"
local len=$prefix_len
for i in {0..3}; do
if (( len >= 8 )); then
mask=255
elif (( len > 0 )); then
mask=$(( 256 - 2 ** ( 8 - len ) ))
else
mask=0
fi
(( len -= 8 ))
result_array[i]=$(( gateway_array[i] & mask ))
done
result="$(printf ".%s" "${result_array[@]}")"
result="${result:1}"
echo "$result/$((32 - prefix_len))"
}
# nat already exists on the Windows VM, the subnet and gateway
# we specify should match that.
gateway="$(powershell -c "(Get-NetIPAddress -InterfaceAlias 'vEthernet (nat)' -AddressFamily IPv4).IPAddress")"
prefix_len="$(powershell -c "(Get-NetIPAddress -InterfaceAlias 'vEthernet (nat)' -AddressFamily IPv4).PrefixLength")"
subnet="$(calculate_subnet "$gateway" "$prefix_len")"
# The "name" field in the config is used as the underlying
# network type right now (see
# https://github.com/microsoft/windows-container-networking/pull/45),
# so it must match a network type in:
# https://docs.microsoft.com/en-us/windows-server/networking/technologies/hcn/hcn-json-document-schemas
bash -c 'cat >"'"${CNI_CONFIG_DIR}"'"/0-containerd-nat.conf <<EOF
{
"cniVersion": "0.2.0",
"name": "nat",
"type": "nat",
"master": "Ethernet",
"ipam": {
"subnet": "'$subnet'",
"routes": [
{
"GW": "'$gateway'"
}
]
},
"capabilities": {
"portMappings": true,
"dns": true
}
}
EOF'

39
hack/install/windows/install-cni.sh Executable file
View File

@ -0,0 +1,39 @@
#!/bin/bash
# Copyright The containerd Authors.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
set -o errexit
set -o nounset
set -o pipefail
source $(dirname "${BASH_SOURCE[0]}")/../utils.sh
# WINCNI_BIN_DIR is the cni plugin directory
WINCNI_BIN_DIR="${WINCNI_BIN_DIR:-"C:\\Program Files\\containerd\\cni\\bin"}"
WINCNI_PKG=github.com/Microsoft/windows-container-networking
WINCNI_VERSION=aa10a0b31e9f72937063436454def1760b858ee2
# Create a temporary GOPATH for cni installation.
GOPATH="$(mktemp -d /tmp/cri-install-cni.XXXX)"
# Install cni
checkout_repo "${WINCNI_PKG}" "${WINCNI_VERSION}" "${WINCNI_PKG}"
cd "${GOPATH}/src/${WINCNI_PKG}"
make all
install -D -m 755 "out/nat.exe" "${WINCNI_BIN_DIR}/nat.exe"
install -D -m 755 "out/sdnbridge.exe" "${WINCNI_BIN_DIR}/sdnbridge.exe"
install -D -m 755 "out/sdnoverlay.exe" "${WINCNI_BIN_DIR}/sdnoverlay.exe"
# Clean the tmp GOPATH dir.
rm -rf "${GOPATH}"

47
hack/install/windows/install-deps.sh Executable file
View File

@ -0,0 +1,47 @@
#!/bin/bash
# Copyright The containerd Authors.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
set -o errexit
set -o nounset
set -o pipefail
cd $(dirname "${BASH_SOURCE[0]}")
# Install hcsshim
./install-hcsshim.sh
# Install cni
./install-cni.sh
# Install cni config
./install-cni-config.sh
# Install containerd
NOSUDO=true \
BUILDTAGS="" \
CONTAINERD_DIR='C:\Program Files\Containerd' \
../install-containerd.sh
# Containerd makefile always installs into a "bin" directory.
# Use slash instead of bach slash so that `*` can work.
mv C:/'Program Files'/Containerd/bin/* 'C:\Program Files\Containerd\'
rm -rf 'C:\Program Files\Containerd\bin'
#Install critools
NOSUDO=true \
CRITOOL_DIR='C:\Program Files\Containerd' \
CRICTL_RUNTIME_ENDPOINT="npipe:////./pipe/containerd-containerd" \
CRICTL_CONFIG_DIR="C:\\Users\\$(id -u -n)\\.crictl" \
../install-critools.sh

37
hack/install/windows/install-hcsshim.sh Executable file
View File

@ -0,0 +1,37 @@
#!/bin/bash
# Copyright The containerd Authors.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
set -o errexit
set -o nounset
set -o pipefail
source $(dirname "${BASH_SOURCE[0]}")/../utils.sh
HCSSHIM_DIR="${HCSSHIM_DIR:-"C:\\Program Files\\Containerd"}"
HCSSHIM_PKG=github.com/Microsoft/hcsshim
# Create a temporary GOPATH for hcsshim installation.
GOPATH="$(mktemp -d /tmp/cri-install-hcsshim.XXXX)"
# Install hcsshim
from-vendor HCSSHIM "${HCSSHIM_PKG}"
checkout_repo "${HCSSHIM_PKG}" "${HCSSHIM_VERSION}" "${HCSSHIM_REPO}"
cd "${GOPATH}/src/${HCSSHIM_PKG}"
go build "${HCSSHIM_PKG}/cmd/containerd-shim-runhcs-v1"
install -D -m 755 containerd-shim-runhcs-v1.exe "${HCSSHIM_DIR}"/containerd-shim-runhcs-v1.exe
# Clean the tmp GOPATH dir. Use sudo because runc build generates
# some privileged files.
rm -rf ${GOPATH}

67
hack/push.sh Executable file
View File

@ -0,0 +1,67 @@
#!/bin/bash
# Copyright The containerd Authors.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
set -o errexit
set -o nounset
set -o pipefail
source $(dirname "${BASH_SOURCE[0]}")/utils.sh
# DEPLOY_BUCKET is the gcs bucket where the tarball should be stored in.
DEPLOY_BUCKET=${DEPLOY_BUCKET:-"cri-containerd-staging"}
# DEPLOY_DIR is the directory in the gcs bucket to store the tarball.
DEPLOY_DIR=${DEPLOY_DIR:-""}
# BUILD_DIR is the directory of the bulid out.
BUILD_DIR=${BUILD_DIR:-"_output"}
# TARBALL is the tarball name.
TARBALL=${TARBALL:-"cri-containerd.tar.gz"}
# LATEST is the name of the latest version file.
LATEST=${LATEST:-"latest"}
# PUSH_VERSION indicates whether to push version.
PUSH_VERSION=${PUSH_VERSION:-false}
release_tar=${ROOT}/${BUILD_DIR}/${TARBALL}
release_tar_checksum=${release_tar}.sha256
if [[ ! -e ${release_tar} || ! -e ${release_tar_checksum} ]]; then
echo "Release tarball is not built"
exit 1
fi
if ! gsutil ls "gs://${DEPLOY_BUCKET}" > /dev/null; then
create_ttl_bucket ${DEPLOY_BUCKET}
fi
if [ -z "${DEPLOY_DIR}" ]; then
DEPLOY_PATH="${DEPLOY_BUCKET}"
else
DEPLOY_PATH="${DEPLOY_BUCKET}/${DEPLOY_DIR}"
fi
# TODO(random-liu): Add checksum for the tarball.
gsutil cp ${release_tar} "gs://${DEPLOY_PATH}/"
gsutil cp ${release_tar_checksum} "gs://${DEPLOY_PATH}/"
echo "Release tarball is uploaded to:
https://storage.googleapis.com/${DEPLOY_PATH}/${TARBALL}"
if ${PUSH_VERSION}; then
if [[ -z "${VERSION}" ]]; then
echo "VERSION is not set"
exit 1
fi
echo ${VERSION} | gsutil cp - "gs://${DEPLOY_PATH}/${LATEST}"
echo "Latest version is uploaded to:
https://storage.googleapis.com/${DEPLOY_PATH}/${LATEST}"
fi

75
hack/release-windows.sh Executable file
View File

@ -0,0 +1,75 @@
#!/bin/bash
# Copyright The containerd Authors.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
set -o errexit
set -o nounset
set -o pipefail
source $(dirname "${BASH_SOURCE[0]}")/utils.sh
cd ${ROOT}
umask 0022
# BUILD_DIR is the directory to generate release tar.
# TARBALL is the name of the release tar.
BUILD_DIR=${BUILD_DIR:-"_output"}
# Convert to absolute path if it's relative.
if [[ ${BUILD_DIR} != /* ]]; then
BUILD_DIR=${ROOT}/${BUILD_DIR}
fi
TARBALL=${TARBALL:-"cri-containerd.tar.gz"}
# INCLUDE_CNI indicates whether to install CNI. By default don't
# include CNI in release tarball.
INCLUDE_CNI=${INCLUDE_CNI:-false}
# CUSTOM_CONTAINERD indicates whether to install customized containerd
# for CI test.
CUSTOM_CONTAINERD=${CUSTOM_CONTAINERD:-false}
destdir=${BUILD_DIR}/release-stage
if [[ -z "${VERSION}" ]]; then
echo "VERSION is not set"
exit 1
fi
# Remove release-stage directory to avoid including old files.
rm -rf ${destdir}
# Install dependencies into release stage.
# Install hcsshim
HCSSHIM_DIR=${destdir} ./hack/install/windows/install-hcsshim.sh
if ${INCLUDE_CNI}; then
# Install cni
NOSUDO=true WINCNI_BIN_DIR=${destdir}/cni ./hack/install/windows/install-cni.sh
fi
# Build containerd from source
NOSUDO=true CONTAINERD_DIR=${destdir} ./hack/install/install-containerd.sh
# Containerd makefile always installs into a "bin" directory.
mv "${destdir}"/bin/* "${destdir}"
rm -rf "${destdir}/bin"
if ${CUSTOM_CONTAINERD}; then
make install -e BINDIR=${destdir}
fi
# Create release tar
tarball=${BUILD_DIR}/${TARBALL}
tar -zcvf ${tarball} -C ${destdir} . --owner=0 --group=0
checksum=$(sha256 ${tarball})
echo "sha256sum: ${checksum} ${tarball}"
echo ${checksum} > ${tarball}.sha256

127
hack/release.sh Executable file
View File

@ -0,0 +1,127 @@
#!/bin/bash
# Copyright The containerd Authors.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
set -o errexit
set -o nounset
set -o pipefail
source $(dirname "${BASH_SOURCE[0]}")/utils.sh
cd ${ROOT}
umask 0022
# BUILD_DIR is the directory to generate release tar.
# TARBALL is the name of the release tar.
BUILD_DIR=${BUILD_DIR:-"_output"}
# Convert to absolute path if it's relative.
if [[ ${BUILD_DIR} != /* ]]; then
BUILD_DIR=${ROOT}/${BUILD_DIR}
fi
TARBALL=${TARBALL:-"cri-containerd.tar.gz"}
# INCLUDE_CNI indicates whether to install CNI. By default don't
# include CNI in release tarball.
INCLUDE_CNI=${INCLUDE_CNI:-false}
# CUSTOM_CONTAINERD indicates whether to install customized containerd
# for CI test.
CUSTOM_CONTAINERD=${CUSTOM_CONTAINERD:-false}
# OFFICIAL_RELEASE indicates whether to use official containerd release.
OFFICIAL_RELEASE=${OFFICIAL_RELEASE:-false}
# LOCAL_RELEASE indicates that containerd has been built and released
# locally.
LOCAL_RELEASE=${LOCAL_RELEASE:-false}
if [ -z "${GOOS:-}" ]
then
GOOS=$(go env GOOS)
fi
if [ -z "${GOARCH:-}" ]
then
GOARCH=$(go env GOARCH)
fi
destdir=${BUILD_DIR}/release-stage
if [[ -z "${VERSION}" ]]; then
echo "VERSION is not set"
exit 1
fi
# Remove release-stage directory to avoid including old files.
rm -rf ${destdir}
# download_containerd downloads containerd from official release.
download_containerd() {
local -r tmppath="$(mktemp -d /tmp/download-containerd.XXXX)"
local -r tarball="${tmppath}/containerd.tar.gz"
local -r url="https://github.com/containerd/containerd/releases/download/v${VERSION}/containerd-${VERSION}.linux-amd64.tar.gz"
wget -O "${tarball}" "${url}"
tar -C "${destdir}/usr/local" -xzf "${tarball}"
rm -rf "${tmppath}"
}
# copy_local_containerd copies local containerd release.
copy_local_containerd() {
local -r tarball="${GOPATH}/src/github.com/containerd/containerd/releases/containerd-${VERSION}.${GOOS}-${GOARCH}.tar.gz"
if [[ ! -e "${tarball}" ]]; then
echo "Containerd release is not built"
exit 1
fi
tar -C "${destdir}/usr/local" -xzf "${tarball}"
}
# Install dependencies into release stage.
# Install runc
NOSUDO=true DESTDIR=${destdir} ./hack/install/install-runc.sh
if ${INCLUDE_CNI}; then
# Install cni
NOSUDO=true DESTDIR=${destdir} ./hack/install/install-cni.sh
fi
# Install critools
NOSUDO=true DESTDIR=${destdir} ./hack/install/install-critools.sh
# Install containerd
if $OFFICIAL_RELEASE; then
download_containerd
elif $LOCAL_RELEASE; then
copy_local_containerd
else
# Build containerd from source
NOSUDO=true DESTDIR=${destdir} ./hack/install/install-containerd.sh
fi
if ${CUSTOM_CONTAINERD}; then
make install -e DESTDIR=${destdir}
fi
# Install systemd units into release stage.
mkdir -p ${destdir}/etc/systemd/system
cp ${ROOT}/contrib/systemd-units/* ${destdir}/etc/systemd/system/
# Install cluster directory into release stage.
mkdir -p ${destdir}/opt/containerd
cp -r ${ROOT}/cluster ${destdir}/opt/containerd
# Write a version file into the release tarball.
cat > ${destdir}/opt/containerd/cluster/version <<EOF
CONTAINERD_VERSION: $(yaml-quote ${VERSION})
EOF
# Create release tar
tarball=${BUILD_DIR}/${TARBALL}
tar -zcvf ${tarball} -C ${destdir} . --owner=0 --group=0
checksum=$(sha256 ${tarball})
echo "sha256sum: ${checksum} ${tarball}"
echo ${checksum} > ${tarball}.sha256

38
hack/sort-vendor.sh Executable file
View File

@ -0,0 +1,38 @@
#!/bin/bash
# Copyright The containerd Authors.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
set -o errexit
set -o nounset
set -o pipefail
source $(dirname "${BASH_SOURCE[0]}")/utils.sh
cd ${ROOT}
echo "Sort vendor.conf..."
tmpdir="$(mktemp -d)"
trap "rm -rf ${tmpdir}" EXIT
awk -v RS= '{print > "'${tmpdir}/'TMP."NR}' vendor.conf
for file in ${tmpdir}/*; do
if [[ -e "${tmpdir}/vendor.conf" ]]; then
echo >> "${tmpdir}/vendor.conf"
fi
sort -Vru "${file}" >> "${tmpdir}/vendor.conf"
done
mv "${tmpdir}/vendor.conf" vendor.conf
echo "Please commit the change made by this file..."

59
hack/sync-vendor.sh Executable file
View File

@ -0,0 +1,59 @@
#!/bin/bash
# Copyright The containerd Authors.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
set -o errexit
set -o nounset
set -o pipefail
source $(dirname "${BASH_SOURCE[0]}")/utils.sh
cd ${ROOT}
echo "Compare vendor with containerd vendors..."
containerd_vendor=$(mktemp /tmp/containerd-vendor.conf.XXXX)
from-vendor CONTAINERD github.com/containerd/containerd
curl -s https://raw.githubusercontent.com/${CONTAINERD_REPO#*/}/${CONTAINERD_VERSION}/vendor.conf > ${containerd_vendor}
# Create a temporary vendor file to update.
tmp_vendor=$(mktemp /tmp/vendor.conf.XXXX)
while read vendor; do
repo=$(echo ${vendor} | awk '{print $1}')
commit=$(echo ${vendor} | awk '{print $2}')
alias=$(echo ${vendor} | awk '{print $3}')
vendor_in_containerd=$(grep ${repo} ${containerd_vendor} || true)
if [ -z "${vendor_in_containerd}" ]; then
echo ${vendor} >> ${tmp_vendor}
continue
fi
commit_in_containerd=$(echo ${vendor_in_containerd} | awk '{print $2}')
alias_in_containerd=$(echo ${vendor_in_containerd} | awk '{print $3}')
if [[ "${commit}" != "${commit_in_containerd}" || "${alias}" != "${alias_in_containerd}" ]]; then
echo ${vendor_in_containerd} >> ${tmp_vendor}
else
echo ${vendor} >> ${tmp_vendor}
fi
done < vendor.conf
# Update vendors if temporary vendor.conf is different from the original one.
if ! diff vendor.conf ${tmp_vendor} > /dev/null; then
if [ $# -gt 0 ] && [ ${1} = "-only-verify" ]; then
echo "Need to update vendor.conf."
diff vendor.conf ${tmp_vendor}
rm ${tmp_vendor}
exit 1
else
echo "Updating vendor.conf."
mv ${tmp_vendor} vendor.conf
fi
fi
rm ${containerd_vendor}

69
hack/test-cri.sh Executable file
View File

@ -0,0 +1,69 @@
#!/bin/bash
# Copyright The containerd Authors.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
set -o nounset
set -o pipefail
source $(dirname "${BASH_SOURCE[0]}")/test-utils.sh
# FOCUS focuses the test to run.
FOCUS=${FOCUS:-}
# SKIP skips the test to skip.
SKIP=${SKIP:-""}
# REPORT_DIR is the the directory to store test logs.
REPORT_DIR=${REPORT_DIR:-"/tmp/test-cri"}
# RUNTIME is the runtime handler to use in the test.
RUNTIME=${RUNTIME:-""}
# Check GOPATH
if [[ -z "${GOPATH}" ]]; then
echo "GOPATH is not set"
exit 1
fi
# For multiple GOPATHs, keep the first one only
GOPATH=${GOPATH%%:*}
CRITEST=${GOPATH}/bin/critest
GINKGO_PKG=github.com/onsi/ginkgo/ginkgo
# Install ginkgo
if [ ! -x "$(command -v ginkgo)" ]; then
go get -u ${GINKGO_PKG}
fi
# Install critest
if [ ! -x "$(command -v ${CRITEST})" ]; then
go get -d ${CRITOOL_PKG}/...
cd ${GOPATH}/src/${CRITOOL_PKG}
git fetch --all
git checkout ${CRITOOL_VERSION}
make critest
make install-critest -e BINDIR="${GOPATH}/bin"
fi
which ${CRITEST}
mkdir -p ${REPORT_DIR}
test_setup ${REPORT_DIR}
# Run cri validation test
sudo env PATH=${PATH} GOPATH=${GOPATH} ${CRITEST} --runtime-endpoint=${CONTAINERD_SOCK} --ginkgo.focus="${FOCUS}" --ginkgo.skip="${SKIP}" --parallel=8 --runtime-handler=${RUNTIME}
test_exit_code=$?
test_teardown
exit ${test_exit_code}

104
hack/test-e2e-node.sh Executable file
View File

@ -0,0 +1,104 @@
#!/bin/bash
# Copyright The containerd Authors.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
set -o nounset
set -o pipefail
source $(dirname "${BASH_SOURCE[0]}")/test-utils.sh
DEFAULT_SKIP="\[Flaky\]|\[Slow\]|\[Serial\]"
DEFAULT_SKIP+="|querying\s\/stats\/summary"
# FOCUS focuses the test to run.
export FOCUS=${FOCUS:-""}
# SKIP skips the test to skip.
export SKIP=${SKIP:-${DEFAULT_SKIP}}
# REPORT_DIR is the the directory to store test logs.
REPORT_DIR=${REPORT_DIR:-"/tmp/test-e2e-node"}
# UPLOAD_LOG indicates whether to upload test log to gcs.
UPLOAD_LOG=${UPLOAD_LOG:-false}
# TIMEOUT is the timeout of the test.
TIMEOUT=${TIMEOUT:-"40m"}
# FAIL_SWAP_ON makes kubelet fail when swap is on.
# Many dev environments run with swap on, so we don't fail by default.
FAIL_SWAP_ON=${FAIL_SWAP_ON:-"false"}
# Check GOPATH
if [[ -z "${GOPATH}" ]]; then
echo "GOPATH is not set"
exit 1
fi
ORIGINAL_RULES=`mktemp`
sudo iptables-save > ${ORIGINAL_RULES}
# Update ip firewall
# We need to add rules to accept all TCP/UDP/ICMP packets.
if sudo iptables -L INPUT | grep "Chain INPUT (policy DROP)" > /dev/null; then
sudo iptables -A INPUT -w -p TCP -j ACCEPT
sudo iptables -A INPUT -w -p UDP -j ACCEPT
sudo iptables -A INPUT -w -p ICMP -j ACCEPT
fi
if sudo iptables -L FORWARD | grep "Chain FORWARD (policy DROP)" > /dev/null; then
sudo iptables -A FORWARD -w -p TCP -j ACCEPT
sudo iptables -A FORWARD -w -p UDP -j ACCEPT
sudo iptables -A FORWARD -w -p ICMP -j ACCEPT
fi
# For multiple GOPATHs, keep the first one only
GOPATH=${GOPATH%%:*}
# Get kubernetes
KUBERNETES_REPO="https://github.com/kubernetes/kubernetes"
KUBERNETES_PATH="${GOPATH}/src/k8s.io/kubernetes"
if [ ! -d "${KUBERNETES_PATH}" ]; then
mkdir -p ${KUBERNETES_PATH}
cd ${KUBERNETES_PATH}
git clone https://${KUBERNETES_REPO} .
fi
cd ${KUBERNETES_PATH}
git fetch --all
git checkout ${KUBERNETES_VERSION}
mkdir -p ${REPORT_DIR}
test_setup ${REPORT_DIR}
timeout "${TIMEOUT}" make test-e2e-node \
RUNTIME=remote \
CONTAINER_RUNTIME_ENDPOINT=unix://${CONTAINERD_SOCK} \
ARTIFACTS=${REPORT_DIR} \
TEST_ARGS='--kubelet-flags=--cgroups-per-qos=true \
--kubelet-flags=--cgroup-root=/ \
--kubelet-flags=--fail-swap-on='${FAIL_SWAP_ON}' \
--prepull-images=false'
test_exit_code=$?
test_teardown
sudo iptables-restore < ${ORIGINAL_RULES}
rm ${ORIGINAL_RULES}
# UPLOAD_LOG_PATH is bucket to upload test logs.
UPLOAD_LOG_PATH=cri-containerd_test-e2e-node
if ${UPLOAD_LOG}; then
if [ -z "${VERSION}" ]; then
echo "VERSION is not set"
exit 1
fi
upload_logs_to_gcs "${UPLOAD_LOG_PATH}" "${VERSION}-$(date +%Y%m%d-%H%M%S)" "${REPORT_DIR}"
fi
exit ${test_exit_code}

46
hack/test-integration.sh Executable file
View File

@ -0,0 +1,46 @@
#!/bin/bash
# Copyright The containerd Authors.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
set -o nounset
set -o pipefail
source $(dirname "${BASH_SOURCE[0]}")/test-utils.sh
cd ${ROOT}
# FOCUS focuses the test to run.
FOCUS=${FOCUS:-""}
# REPORT_DIR is the the directory to store test logs.
REPORT_DIR=${REPORT_DIR:-"/tmp/test-integration"}
# RUNTIME is the runtime handler to use in the test.
RUNTIME=${RUNTIME:-""}
CRI_ROOT="${CONTAINERD_ROOT}/io.containerd.grpc.v1.cri"
mkdir -p ${REPORT_DIR}
test_setup ${REPORT_DIR}
# Run integration test.
sudo PATH=${PATH} ${ROOT}/_output/integration.test --test.run="${FOCUS}" --test.v \
--cri-endpoint=${CONTAINERD_SOCK} \
--cri-root=${CRI_ROOT} \
--runtime-handler=${RUNTIME} \
--containerd-bin=${CONTAINERD_BIN}
test_exit_code=$?
test_teardown
exit ${test_exit_code}

119
hack/test-utils.sh Executable file
View File

@ -0,0 +1,119 @@
#!/bin/bash
# Copyright The containerd Authors.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
source $(dirname "${BASH_SOURCE[0]}")/utils.sh
# RESTART_WAIT_PERIOD is the period to wait before restarting containerd.
RESTART_WAIT_PERIOD=${RESTART_WAIT_PERIOD:-10}
# CONTAINERD_FLAGS contains all containerd flags.
CONTAINERD_FLAGS="--log-level=debug "
# Use a configuration file for containerd.
CONTAINERD_CONFIG_FILE=${CONTAINERD_CONFIG_FILE:-""}
if [ -z "${CONTAINERD_CONFIG_FILE}" ] && command -v sestatus >/dev/null 2>&1; then
selinux_config="/tmp/containerd-config-selinux.toml"
cat >${selinux_config} <<<'
[plugins.cri]
enable_selinux = true
'
CONTAINERD_CONFIG_FILE=${CONTAINERD_CONFIG_FILE:-"${selinux_config}"}
fi
# CONTAINERD_TEST_SUFFIX is the suffix appended to the root/state directory used
# by test containerd.
CONTAINERD_TEST_SUFFIX=${CONTAINERD_TEST_SUFFIX:-"-test"}
# The containerd root directory.
CONTAINERD_ROOT=${CONTAINERD_ROOT:-"/var/lib/containerd${CONTAINERD_TEST_SUFFIX}"}
# The containerd state directory.
CONTAINERD_STATE=${CONTAINERD_STATE:-"/run/containerd${CONTAINERD_TEST_SUFFIX}"}
# The containerd socket address.
CONTAINERD_SOCK=${CONTAINERD_SOCK:-unix://${CONTAINERD_STATE}/containerd.sock}
# The containerd binary name.
CONTAINERD_BIN=${CONTAINERD_BIN:-"containerd${CONTAINERD_TEST_SUFFIX}"}
if [ -f "${CONTAINERD_CONFIG_FILE}" ]; then
CONTAINERD_FLAGS+="--config ${CONTAINERD_CONFIG_FILE} "
fi
CONTAINERD_FLAGS+="--address ${CONTAINERD_SOCK#"unix://"} \
--state ${CONTAINERD_STATE} \
--root ${CONTAINERD_ROOT}"
containerd_groupid=
# test_setup starts containerd.
test_setup() {
local report_dir=$1
# Start containerd
if [ ! -x "${ROOT}/_output/containerd" ]; then
echo "containerd is not built"
exit 1
fi
# rename the test containerd binary, so that we can easily
# distinguish it.
cp ${ROOT}/_output/containerd ${ROOT}/_output/${CONTAINERD_BIN}
set -m
# Create containerd in a different process group
# so that we can easily clean them up.
keepalive "sudo PATH=${PATH} ${ROOT}/_output/${CONTAINERD_BIN} ${CONTAINERD_FLAGS}" \
${RESTART_WAIT_PERIOD} &> ${report_dir}/containerd.log &
pid=$!
set +m
containerd_groupid=$(ps -o pgid= -p ${pid})
# Wait for containerd to be running by using the containerd client ctr to check the version
# of the containerd server. Wait an increasing amount of time after each of five attempts
local -r ctr_path=$(which ctr)
if [ -z "${ctr_path}" ]; then
echo "ctr is not in PATH"
exit 1
fi
local -r crictl_path=$(which crictl)
if [ -z "${crictl_path}" ]; then
echo "crictl is not in PATH"
exit 1
fi
readiness_check "sudo ${ctr_path} --address ${CONTAINERD_SOCK#"unix://"} version"
readiness_check "sudo ${crictl_path} --runtime-endpoint=${CONTAINERD_SOCK} info"
}
# test_teardown kills containerd.
test_teardown() {
if [ -n "${containerd_groupid}" ]; then
sudo pkill -g ${containerd_groupid}
fi
}
# keepalive runs a command and keeps it alive.
# keepalive process is eventually killed in test_teardown.
keepalive() {
local command=$1
echo ${command}
local wait_period=$2
while true; do
${command}
sleep ${wait_period}
done
}
# readiness_check checks readiness of a daemon with specified command.
readiness_check() {
local command=$1
local MAX_ATTEMPTS=5
local attempt_num=1
until ${command} &> /dev/null || (( attempt_num == MAX_ATTEMPTS ))
do
echo "$attempt_num attempt \"$command\"! Trying again in $attempt_num seconds..."
sleep $(( attempt_num++ ))
done
}

44
hack/update-proto.sh Executable file
View File

@ -0,0 +1,44 @@
#!/bin/bash
# Copyright The containerd Authors.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
set -o errexit
set -o nounset
set -o pipefail
ROOT="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"/..
API_ROOT="${ROOT}/${API_PATH-"pkg/api/v1"}"
go get k8s.io/code-generator/cmd/go-to-protobuf/protoc-gen-gogo
if ! which protoc-gen-gogo >/dev/null; then
echo "GOPATH is not in PATH"
exit 1
fi
function cleanup {
rm -f ${API_ROOT}/api.pb.go.bak
}
trap cleanup EXIT
protoc \
--proto_path="${API_ROOT}" \
--proto_path="${ROOT}/vendor" \
--gogo_out=plugins=grpc:${API_ROOT} ${API_ROOT}/api.proto
# Update boilerplate for the generated file.
echo "$(cat hack/boilerplate/boilerplate ${API_ROOT}/api.pb.go)" > ${API_ROOT}/api.pb.go
gofmt -l -s -w ${API_ROOT}/api.pb.go

105
hack/utils.sh Executable file
View File

@ -0,0 +1,105 @@
#!/bin/bash
# Copyright The containerd Authors.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
ROOT="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"/..
# Not from vendor.conf.
KUBERNETES_VERSION="v1.19.0-beta.2"
CRITOOL_VERSION=${CRITOOL_VERSION:-baca4a152dfe671fc17911a7af74bcb61680ee39}
CRITOOL_PKG=github.com/kubernetes-sigs/cri-tools
CRITOOL_REPO=github.com/kubernetes-sigs/cri-tools
# VENDOR is the path to vendor.conf.
VENDOR=${VENDOR:-"${ROOT}/vendor.conf"}
# upload_logs_to_gcs uploads test logs to gcs.
# Var set:
# 1. Bucket: gcs bucket to upload logs.
# 2. Dir: directory name to upload logs.
# 3. Test Result: directory of the test result.
upload_logs_to_gcs() {
local -r bucket=$1
local -r dir=$2
local -r result=$3
if ! gsutil ls "gs://${bucket}" > /dev/null; then
create_ttl_bucket ${bucket}
fi
local -r upload_log_path=${bucket}/${dir}
gsutil cp -r "${result}" "gs://${upload_log_path}"
echo "Test logs are uploaed to:
http://gcsweb.k8s.io/gcs/${upload_log_path}/"
}
# create_ttl_bucket create a public bucket in which all objects
# have a default TTL (30 days).
# Var set:
# 1. Bucket: gcs bucket name.
create_ttl_bucket() {
local -r bucket=$1
gsutil mb "gs://${bucket}"
local -r bucket_rule=$(mktemp)
# Set 30 day TTL for logs inside the bucket.
echo '{"rule": [{"action": {"type": "Delete"},"condition": {"age": 30}}]}' > ${bucket_rule}
gsutil lifecycle set "${bucket_rule}" "gs://${bucket}"
rm "${bucket_rule}"
gsutil -m acl ch -g all:R "gs://${bucket}"
gsutil defacl set public-read "gs://${bucket}"
}
# sha256 generates a sha256 checksum for a file.
# Var set:
# 1. Filename.
sha256() {
if which sha256sum >/dev/null 2>&1; then
sha256sum "$1" | awk '{ print $1 }'
else
shasum -a256 "$1" | awk '{ print $1 }'
fi
}
# Takes a prefix ($what) and a $repo and sets `$what_VERSION` and
# `$what_REPO` from vendor.conf, where `$what_REPO` defaults to $repo
# but is overridden by the 3rd field of vendor.conf.
from-vendor() {
local what=$1
local repo=$2
local vendor=$VENDOR
setvars=$(awk -v REPO=$repo -v WHAT=$what -- '
BEGIN { rc=1 } # Assume we did not find what we were looking for.
// {
if ($1 == REPO) {
if ($3 != "" && $3 !~ /#.*/ ) { gsub(/http.*\/\//, "", $3); REPO = $3 }; # Override repo.
printf("%s_VERSION=%s; %s_REPO=%s\n", WHAT, $2, WHAT, REPO);
rc=0; # Note success for use in END block.
exit # No point looking further.
}
}
END { exit rc } # Exit with the desired code.
' $vendor)
if [ $? -ne 0 ] ; then
echo "failed to get version of $repo from $vendor" >&2
exit 1
fi
eval $setvars
}
# yaml-quote quotes something appropriate for a yaml string.
# This is the same with:
# https://github.com/kubernetes/kubernetes/blob/v1.10.1/cluster/gce/util.sh#L471.
yaml-quote() {
echo "'$(echo "${@:-}" | sed -e "s/'/''/g")'"
}

35
hack/verify-gofmt.sh Executable file
View File

@ -0,0 +1,35 @@
#!/bin/bash
# Copyright The containerd Authors.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
set -o errexit
set -o nounset
set -o pipefail
find_files() {
find . -not \( \
\( \
-wholename '*/vendor/*' \
\) -prune \
\) -name '*.go'
}
GOFMT="gofmt -s"
bad_files=$(find_files | xargs $GOFMT -l)
if [[ -n "${bad_files}" ]]; then
echo "!!! '$GOFMT' needs to be run on the following files: "
echo "${bad_files}"
exit 1
fi

25
hack/verify-vendor.sh Executable file
View File

@ -0,0 +1,25 @@
#!/usr/bin/env bash
# Copyright The containerd Authors.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
set -o errexit
set -o nounset
set -o pipefail
tmpdir="$(mktemp -d)"
trap "rm -rf ${tmpdir}" EXIT
git clone "https://github.com/containerd/project" "${tmpdir}"
"${tmpdir}"/script/validate/vendor

89
integration/addition_gids_test.go Normal file
View File

@ -0,0 +1,89 @@
// +build linux
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package integration
import (
"io/ioutil"
"os"
"path/filepath"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
runtime "k8s.io/cri-api/pkg/apis/runtime/v1alpha2"
)
func TestAdditionalGids(t *testing.T) {
testPodLogDir, err := ioutil.TempDir("/tmp", "additional-gids")
require.NoError(t, err)
defer os.RemoveAll(testPodLogDir)
t.Log("Create a sandbox with log directory")
sbConfig := PodSandboxConfig("sandbox", "additional-gids",
WithPodLogDirectory(testPodLogDir))
sb, err := runtimeService.RunPodSandbox(sbConfig, *runtimeHandler)
require.NoError(t, err)
defer func() {
assert.NoError(t, runtimeService.StopPodSandbox(sb))
assert.NoError(t, runtimeService.RemovePodSandbox(sb))
}()
const (
testImage = "busybox"
containerName = "test-container"
)
t.Logf("Pull test image %q", testImage)
img, err := imageService.PullImage(&runtime.ImageSpec{Image: testImage}, nil, sbConfig)
require.NoError(t, err)
defer func() {
assert.NoError(t, imageService.RemoveImage(&runtime.ImageSpec{Image: img}))
}()
t.Log("Create a container to print id")
cnConfig := ContainerConfig(
containerName,
"busybox",
WithCommand("id"),
WithLogPath(containerName),
WithSupplementalGroups([]int64{1 /*daemon*/, 1234 /*new group*/}),
)
cn, err := runtimeService.CreateContainer(sb, cnConfig, sbConfig)
require.NoError(t, err)
t.Log("Start the container")
require.NoError(t, runtimeService.StartContainer(cn))
t.Log("Wait for container to finish running")
require.NoError(t, Eventually(func() (bool, error) {
s, err := runtimeService.ContainerStatus(cn)
if err != nil {
return false, err
}
if s.GetState() == runtime.ContainerState_CONTAINER_EXITED {
return true, nil
}
return false, nil
}, time.Second, 30*time.Second))
t.Log("Search additional groups in container log")
content, err := ioutil.ReadFile(filepath.Join(testPodLogDir, containerName))
assert.NoError(t, err)
assert.Contains(t, string(content), "groups=1(daemon),10(wheel),1234")
}

175
integration/container_log_test.go Normal file
View File

@ -0,0 +1,175 @@
// +build linux
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package integration
import (
"fmt"
"io/ioutil"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
runtime "k8s.io/cri-api/pkg/apis/runtime/v1alpha2"
)
func TestContainerLogWithoutTailingNewLine(t *testing.T) {
testPodLogDir, err := ioutil.TempDir("/tmp", "container-log-without-tailing-newline")
require.NoError(t, err)
defer os.RemoveAll(testPodLogDir)
t.Log("Create a sandbox with log directory")
sbConfig := PodSandboxConfig("sandbox", "container-log-without-tailing-newline",
WithPodLogDirectory(testPodLogDir),
)
sb, err := runtimeService.RunPodSandbox(sbConfig, *runtimeHandler)
require.NoError(t, err)
defer func() {
assert.NoError(t, runtimeService.StopPodSandbox(sb))
assert.NoError(t, runtimeService.RemovePodSandbox(sb))
}()
const (
testImage = "busybox"
containerName = "test-container"
)
t.Logf("Pull test image %q", testImage)
img, err := imageService.PullImage(&runtime.ImageSpec{Image: testImage}, nil, sbConfig)
require.NoError(t, err)
defer func() {
assert.NoError(t, imageService.RemoveImage(&runtime.ImageSpec{Image: img}))
}()
t.Log("Create a container with log path")
cnConfig := ContainerConfig(
containerName,
testImage,
WithCommand("sh", "-c", "printf abcd"),
WithLogPath(containerName),
)
cn, err := runtimeService.CreateContainer(sb, cnConfig, sbConfig)
require.NoError(t, err)
t.Log("Start the container")
require.NoError(t, runtimeService.StartContainer(cn))
t.Log("Wait for container to finish running")
require.NoError(t, Eventually(func() (bool, error) {
s, err := runtimeService.ContainerStatus(cn)
if err != nil {
return false, err
}
if s.GetState() == runtime.ContainerState_CONTAINER_EXITED {
return true, nil
}
return false, nil
}, time.Second, 30*time.Second))
t.Log("Check container log")
content, err := ioutil.ReadFile(filepath.Join(testPodLogDir, containerName))
assert.NoError(t, err)
checkContainerLog(t, string(content), []string{
fmt.Sprintf("%s %s %s", runtime.Stdout, runtime.LogTagPartial, "abcd"),
})
}
func TestLongContainerLog(t *testing.T) {
testPodLogDir, err := ioutil.TempDir("/tmp", "long-container-log")
require.NoError(t, err)
defer os.RemoveAll(testPodLogDir)
t.Log("Create a sandbox with log directory")
sbConfig := PodSandboxConfig("sandbox", "long-container-log",
WithPodLogDirectory(testPodLogDir),
)
sb, err := runtimeService.RunPodSandbox(sbConfig, *runtimeHandler)
require.NoError(t, err)
defer func() {
assert.NoError(t, runtimeService.StopPodSandbox(sb))
assert.NoError(t, runtimeService.RemovePodSandbox(sb))
}()
const (
testImage = "busybox"
containerName = "test-container"
)
t.Logf("Pull test image %q", testImage)
img, err := imageService.PullImage(&runtime.ImageSpec{Image: testImage}, nil, sbConfig)
require.NoError(t, err)
defer func() {
assert.NoError(t, imageService.RemoveImage(&runtime.ImageSpec{Image: img}))
}()
t.Log("Create a container with log path")
config, err := CRIConfig()
require.NoError(t, err)
maxSize := config.MaxContainerLogLineSize
shortLineCmd := fmt.Sprintf("i=0; while [ $i -lt %d ]; do printf %s; i=$((i+1)); done", maxSize-1, "a")
maxLenLineCmd := fmt.Sprintf("i=0; while [ $i -lt %d ]; do printf %s; i=$((i+1)); done", maxSize, "b")
longLineCmd := fmt.Sprintf("i=0; while [ $i -lt %d ]; do printf %s; i=$((i+1)); done", maxSize+1, "c")
cnConfig := ContainerConfig(
containerName,
testImage,
WithCommand("sh", "-c",
fmt.Sprintf("%s; echo; %s; echo; %s; echo", shortLineCmd, maxLenLineCmd, longLineCmd)),
WithLogPath(containerName),
)
cn, err := runtimeService.CreateContainer(sb, cnConfig, sbConfig)
require.NoError(t, err)
t.Log("Start the container")
require.NoError(t, runtimeService.StartContainer(cn))
t.Log("Wait for container to finish running")
require.NoError(t, Eventually(func() (bool, error) {
s, err := runtimeService.ContainerStatus(cn)
if err != nil {
return false, err
}
if s.GetState() == runtime.ContainerState_CONTAINER_EXITED {
return true, nil
}
return false, nil
}, time.Second, 30*time.Second))
t.Log("Check container log")
content, err := ioutil.ReadFile(filepath.Join(testPodLogDir, containerName))
assert.NoError(t, err)
checkContainerLog(t, string(content), []string{
fmt.Sprintf("%s %s %s", runtime.Stdout, runtime.LogTagFull, strings.Repeat("a", maxSize-1)),
fmt.Sprintf("%s %s %s", runtime.Stdout, runtime.LogTagFull, strings.Repeat("b", maxSize)),
fmt.Sprintf("%s %s %s", runtime.Stdout, runtime.LogTagPartial, strings.Repeat("c", maxSize)),
fmt.Sprintf("%s %s %s", runtime.Stdout, runtime.LogTagFull, "c"),
})
}
func checkContainerLog(t *testing.T, log string, messages []string) {
lines := strings.Split(strings.TrimSpace(log), "\n")
require.Len(t, lines, len(messages), "log line number should match")
for i, line := range lines {
parts := strings.SplitN(line, " ", 2)
require.Len(t, parts, 2)
_, err := time.Parse(time.RFC3339Nano, parts[0])
assert.NoError(t, err, "timestamp should be in RFC3339Nano format")
assert.Equal(t, messages[i], parts[1], "log content should match")
}
}

62
integration/container_restart_test.go Normal file
View File

@ -0,0 +1,62 @@
// +build linux
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package integration
import (
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// Test to verify container can be restarted
func TestContainerRestart(t *testing.T) {
t.Logf("Create a pod config and run sandbox container")
sbConfig := PodSandboxConfig("sandbox1", "restart")
sb, err := runtimeService.RunPodSandbox(sbConfig, *runtimeHandler)
require.NoError(t, err)
defer func() {
assert.NoError(t, runtimeService.StopPodSandbox(sb))
assert.NoError(t, runtimeService.RemovePodSandbox(sb))
}()
t.Logf("Create a container config and run container in a pod")
containerConfig := ContainerConfig(
"container1",
pauseImage,
WithTestLabels(),
WithTestAnnotations(),
)
cn, err := runtimeService.CreateContainer(sb, containerConfig, sbConfig)
require.NoError(t, err)
defer func() {
assert.NoError(t, runtimeService.RemoveContainer(cn))
}()
require.NoError(t, runtimeService.StartContainer(cn))
defer func() {
assert.NoError(t, runtimeService.StopContainer(cn, 10))
}()
t.Logf("Restart the container with same config")
require.NoError(t, runtimeService.StopContainer(cn, 10))
require.NoError(t, runtimeService.RemoveContainer(cn))
cn, err = runtimeService.CreateContainer(sb, containerConfig, sbConfig)
require.NoError(t, err)
require.NoError(t, runtimeService.StartContainer(cn))
}

347
integration/container_stats_test.go Normal file
View File

@ -0,0 +1,347 @@
// +build linux
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package integration
import (
"fmt"
"testing"
"time"
"github.com/pkg/errors"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
runtime "k8s.io/cri-api/pkg/apis/runtime/v1alpha2"
)
// Test to verify for a container ID
func TestContainerStats(t *testing.T) {
t.Logf("Create a pod config and run sandbox container")
sbConfig := PodSandboxConfig("sandbox1", "stats")
sb, err := runtimeService.RunPodSandbox(sbConfig, *runtimeHandler)
require.NoError(t, err)
defer func() {
assert.NoError(t, runtimeService.StopPodSandbox(sb))
assert.NoError(t, runtimeService.RemovePodSandbox(sb))
}()
t.Logf("Create a container config and run container in a pod")
containerConfig := ContainerConfig(
"container1",
pauseImage,
WithTestLabels(),
WithTestAnnotations(),
)
cn, err := runtimeService.CreateContainer(sb, containerConfig, sbConfig)
require.NoError(t, err)
defer func() {
assert.NoError(t, runtimeService.RemoveContainer(cn))
}()
require.NoError(t, runtimeService.StartContainer(cn))
defer func() {
assert.NoError(t, runtimeService.StopContainer(cn, 10))
}()
t.Logf("Fetch stats for container")
var s *runtime.ContainerStats
require.NoError(t, Eventually(func() (bool, error) {
s, err = runtimeService.ContainerStats(cn)
if err != nil {
return false, err
}
if s.GetWritableLayer().GetUsedBytes().GetValue() != 0 &&
s.GetWritableLayer().GetInodesUsed().GetValue() != 0 {
return true, nil
}
return false, nil
}, time.Second, 30*time.Second))
t.Logf("Verify stats received for container %q", cn)
testStats(t, s, containerConfig)
}
// Test to verify filtering without any filter
func TestContainerListStats(t *testing.T) {
t.Logf("Create a pod config and run sandbox container")
sbConfig := PodSandboxConfig("running-pod", "statsls")
sb, err := runtimeService.RunPodSandbox(sbConfig, *runtimeHandler)
require.NoError(t, err)
defer func() {
assert.NoError(t, runtimeService.StopPodSandbox(sb))
assert.NoError(t, runtimeService.RemovePodSandbox(sb))
}()
t.Logf("Create a container config and run containers in a pod")
containerConfigMap := make(map[string]*runtime.ContainerConfig)
for i := 0; i < 3; i++ {
cName := fmt.Sprintf("container%d", i)
containerConfig := ContainerConfig(
cName,
pauseImage,
WithTestLabels(),
WithTestAnnotations(),
)
cn, err := runtimeService.CreateContainer(sb, containerConfig, sbConfig)
require.NoError(t, err)
containerConfigMap[cn] = containerConfig
defer func() {
assert.NoError(t, runtimeService.RemoveContainer(cn))
}()
require.NoError(t, runtimeService.StartContainer(cn))
defer func() {
assert.NoError(t, runtimeService.StopContainer(cn, 10))
}()
}
t.Logf("Fetch all container stats")
var stats []*runtime.ContainerStats
require.NoError(t, Eventually(func() (bool, error) {
stats, err = runtimeService.ListContainerStats(&runtime.ContainerStatsFilter{})
if err != nil {
return false, err
}
for _, s := range stats {
if s.GetWritableLayer().GetUsedBytes().GetValue() == 0 &&
s.GetWritableLayer().GetInodesUsed().GetValue() == 0 {
return false, nil
}
}
return true, nil
}, time.Second, 30*time.Second))
t.Logf("Verify all container stats")
for _, s := range stats {
testStats(t, s, containerConfigMap[s.GetAttributes().GetId()])
}
}
// Test to verify filtering given a specific container ID
// TODO Convert the filter tests into table driven tests and unit tests
func TestContainerListStatsWithIdFilter(t *testing.T) {
t.Logf("Create a pod config and run sandbox container")
sbConfig := PodSandboxConfig("running-pod", "statsls")
sb, err := runtimeService.RunPodSandbox(sbConfig, *runtimeHandler)
require.NoError(t, err)
defer func() {
assert.NoError(t, runtimeService.StopPodSandbox(sb))
assert.NoError(t, runtimeService.RemovePodSandbox(sb))
}()
t.Logf("Create a container config and run containers in a pod")
containerConfigMap := make(map[string]*runtime.ContainerConfig)
for i := 0; i < 3; i++ {
cName := fmt.Sprintf("container%d", i)
containerConfig := ContainerConfig(
cName,
pauseImage,
WithTestLabels(),
WithTestAnnotations(),
)
cn, err := runtimeService.CreateContainer(sb, containerConfig, sbConfig)
containerConfigMap[cn] = containerConfig
require.NoError(t, err)
defer func() {
assert.NoError(t, runtimeService.RemoveContainer(cn))
}()
require.NoError(t, runtimeService.StartContainer(cn))
defer func() {
assert.NoError(t, runtimeService.StopContainer(cn, 10))
}()
}
t.Logf("Fetch container stats for each container with Filter")
var stats []*runtime.ContainerStats
for id := range containerConfigMap {
require.NoError(t, Eventually(func() (bool, error) {
stats, err = runtimeService.ListContainerStats(
&runtime.ContainerStatsFilter{Id: id})
if err != nil {
return false, err
}
if len(stats) != 1 {
return false, errors.New("unexpected stats length")
}
if stats[0].GetWritableLayer().GetUsedBytes().GetValue() != 0 &&
stats[0].GetWritableLayer().GetInodesUsed().GetValue() != 0 {
return true, nil
}
return false, nil
}, time.Second, 30*time.Second))
t.Logf("Verify container stats for %s", id)
for _, s := range stats {
require.Equal(t, s.GetAttributes().GetId(), id)
testStats(t, s, containerConfigMap[id])
}
}
}
// Test to verify filtering given a specific Sandbox ID. Stats for
// all the containers in a pod should be returned
func TestContainerListStatsWithSandboxIdFilter(t *testing.T) {
t.Logf("Create a pod config and run sandbox container")
sbConfig := PodSandboxConfig("running-pod", "statsls")
sb, err := runtimeService.RunPodSandbox(sbConfig, *runtimeHandler)
require.NoError(t, err)
defer func() {
assert.NoError(t, runtimeService.StopPodSandbox(sb))
assert.NoError(t, runtimeService.RemovePodSandbox(sb))
}()
t.Logf("Create a container config and run containers in a pod")
containerConfigMap := make(map[string]*runtime.ContainerConfig)
for i := 0; i < 3; i++ {
cName := fmt.Sprintf("container%d", i)
containerConfig := ContainerConfig(
cName,
pauseImage,
WithTestLabels(),
WithTestAnnotations(),
)
cn, err := runtimeService.CreateContainer(sb, containerConfig, sbConfig)
containerConfigMap[cn] = containerConfig
require.NoError(t, err)
defer func() {
assert.NoError(t, runtimeService.RemoveContainer(cn))
}()
require.NoError(t, runtimeService.StartContainer(cn))
defer func() {
assert.NoError(t, runtimeService.StopContainer(cn, 10))
}()
}
t.Logf("Fetch container stats for each container with Filter")
var stats []*runtime.ContainerStats
require.NoError(t, Eventually(func() (bool, error) {
stats, err = runtimeService.ListContainerStats(
&runtime.ContainerStatsFilter{PodSandboxId: sb})
if err != nil {
return false, err
}
if len(stats) != 3 {
return false, errors.New("unexpected stats length")
}
if stats[0].GetWritableLayer().GetUsedBytes().GetValue() != 0 &&
stats[0].GetWritableLayer().GetInodesUsed().GetValue() != 0 {
return true, nil
}
return false, nil
}, time.Second, 30*time.Second))
t.Logf("Verify container stats for sandbox %q", sb)
for _, s := range stats {
testStats(t, s, containerConfigMap[s.GetAttributes().GetId()])
}
}
// Test to verify filtering given a specific container ID and
// sandbox ID
func TestContainerListStatsWithIdSandboxIdFilter(t *testing.T) {
t.Logf("Create a pod config and run sandbox container")
sbConfig := PodSandboxConfig("running-pod", "statsls")
sb, err := runtimeService.RunPodSandbox(sbConfig, *runtimeHandler)
require.NoError(t, err)
defer func() {
assert.NoError(t, runtimeService.StopPodSandbox(sb))
assert.NoError(t, runtimeService.RemovePodSandbox(sb))
}()
t.Logf("Create container config and run containers in a pod")
containerConfigMap := make(map[string]*runtime.ContainerConfig)
for i := 0; i < 3; i++ {
cName := fmt.Sprintf("container%d", i)
containerConfig := ContainerConfig(
cName,
pauseImage,
WithTestLabels(),
WithTestAnnotations(),
)
cn, err := runtimeService.CreateContainer(sb, containerConfig, sbConfig)
containerConfigMap[cn] = containerConfig
require.NoError(t, err)
defer func() {
assert.NoError(t, runtimeService.RemoveContainer(cn))
}()
require.NoError(t, runtimeService.StartContainer(cn))
defer func() {
assert.NoError(t, runtimeService.StopContainer(cn, 10))
}()
}
t.Logf("Fetch container stats for sandbox ID and container ID filter")
var stats []*runtime.ContainerStats
for id, config := range containerConfigMap {
require.NoError(t, Eventually(func() (bool, error) {
stats, err = runtimeService.ListContainerStats(
&runtime.ContainerStatsFilter{Id: id, PodSandboxId: sb})
if err != nil {
return false, err
}
if len(stats) != 1 {
return false, errors.New("unexpected stats length")
}
if stats[0].GetWritableLayer().GetUsedBytes().GetValue() != 0 &&
stats[0].GetWritableLayer().GetInodesUsed().GetValue() != 0 {
return true, nil
}
return false, nil
}, time.Second, 30*time.Second))
t.Logf("Verify container stats for sandbox %q and container %q filter", sb, id)
for _, s := range stats {
testStats(t, s, config)
}
}
t.Logf("Fetch container stats for sandbox truncID and container truncID filter ")
for id, config := range containerConfigMap {
require.NoError(t, Eventually(func() (bool, error) {
stats, err = runtimeService.ListContainerStats(
&runtime.ContainerStatsFilter{Id: id[:3], PodSandboxId: sb[:3]})
if err != nil {
return false, err
}
if len(stats) != 1 {
return false, errors.New("unexpected stats length")
}
if stats[0].GetWritableLayer().GetUsedBytes().GetValue() != 0 &&
stats[0].GetWritableLayer().GetInodesUsed().GetValue() != 0 {
return true, nil
}
return false, nil
}, time.Second, 30*time.Second))
t.Logf("Verify container stats for sandbox %q and container %q filter", sb, id)
for _, s := range stats {
testStats(t, s, config)
}
}
}
// TODO make this as options to use for dead container tests
func testStats(t *testing.T,
s *runtime.ContainerStats,
config *runtime.ContainerConfig,
) {
require.NotEmpty(t, s.GetAttributes().GetId())
require.NotEmpty(t, s.GetAttributes().GetMetadata())
require.NotEmpty(t, s.GetAttributes().GetAnnotations())
require.Equal(t, s.GetAttributes().GetLabels(), config.Labels)
require.Equal(t, s.GetAttributes().GetAnnotations(), config.Annotations)
require.Equal(t, s.GetAttributes().GetMetadata().Name, config.Metadata.Name)
require.NotEmpty(t, s.GetAttributes().GetLabels())
require.NotEmpty(t, s.GetCpu().GetTimestamp())
require.NotEmpty(t, s.GetCpu().GetUsageCoreNanoSeconds().GetValue())
require.NotEmpty(t, s.GetMemory().GetTimestamp())
require.NotEmpty(t, s.GetMemory().GetWorkingSetBytes().GetValue())
require.NotEmpty(t, s.GetWritableLayer().GetTimestamp())
require.NotEmpty(t, s.GetWritableLayer().GetFsId().GetMountpoint())
require.NotEmpty(t, s.GetWritableLayer().GetUsedBytes().GetValue())
require.NotEmpty(t, s.GetWritableLayer().GetInodesUsed().GetValue())
}

141
integration/container_stop_test.go Normal file
View File

@ -0,0 +1,141 @@
// +build linux
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package integration
import (
"context"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
runtime "k8s.io/cri-api/pkg/apis/runtime/v1alpha2"
)
func TestSharedPidMultiProcessContainerStop(t *testing.T) {
for name, sbConfig := range map[string]*runtime.PodSandboxConfig{
"hostpid": PodSandboxConfig("sandbox", "host-pid-container-stop", WithHostPid),
"podpid": PodSandboxConfig("sandbox", "pod-pid-container-stop", WithPodPid),
} {
t.Run(name, func(t *testing.T) {
t.Log("Create a shared pid sandbox")
sb, err := runtimeService.RunPodSandbox(sbConfig, *runtimeHandler)
require.NoError(t, err)
defer func() {
assert.NoError(t, runtimeService.StopPodSandbox(sb))
assert.NoError(t, runtimeService.RemovePodSandbox(sb))
}()
const (
testImage = "busybox"
containerName = "test-container"
)
t.Logf("Pull test image %q", testImage)
img, err := imageService.PullImage(&runtime.ImageSpec{Image: testImage}, nil, sbConfig)
require.NoError(t, err)
defer func() {
assert.NoError(t, imageService.RemoveImage(&runtime.ImageSpec{Image: img}))
}()
t.Log("Create a multi-process container")
cnConfig := ContainerConfig(
containerName,
testImage,
WithCommand("sh", "-c", "sleep 10000 & sleep 10000"),
)
cn, err := runtimeService.CreateContainer(sb, cnConfig, sbConfig)
require.NoError(t, err)
t.Log("Start the container")
require.NoError(t, runtimeService.StartContainer(cn))
t.Log("Stop the container")
require.NoError(t, runtimeService.StopContainer(cn, 0))
t.Log("The container state should be exited")
s, err := runtimeService.ContainerStatus(cn)
require.NoError(t, err)
assert.Equal(t, s.GetState(), runtime.ContainerState_CONTAINER_EXITED)
})
}
}
func TestContainerStopCancellation(t *testing.T) {
t.Log("Create a pod sandbox")
sbConfig := PodSandboxConfig("sandbox", "cancel-container-stop")
sb, err := runtimeService.RunPodSandbox(sbConfig, *runtimeHandler)
require.NoError(t, err)
defer func() {
assert.NoError(t, runtimeService.StopPodSandbox(sb))
assert.NoError(t, runtimeService.RemovePodSandbox(sb))
}()
const (
testImage = "busybox"
containerName = "test-container"
)
t.Logf("Pull test image %q", testImage)
img, err := imageService.PullImage(&runtime.ImageSpec{Image: testImage}, nil, sbConfig)
require.NoError(t, err)
defer func() {
assert.NoError(t, imageService.RemoveImage(&runtime.ImageSpec{Image: img}))
}()
t.Log("Create a container which traps sigterm")
cnConfig := ContainerConfig(
containerName,
testImage,
WithCommand("sh", "-c", `trap "echo ignore sigterm" TERM; sleep 1000`),
)
cn, err := runtimeService.CreateContainer(sb, cnConfig, sbConfig)
require.NoError(t, err)
t.Log("Start the container")
require.NoError(t, runtimeService.StartContainer(cn))
t.Log("Stop the container with 3s timeout, but 1s context timeout")
// Note that with container pid namespace, the sleep process
// is pid 1, and SIGTERM sent by `StopContainer` will be ignored.
rawClient, err := RawRuntimeClient()
require.NoError(t, err)
ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
defer cancel()
_, err = rawClient.StopContainer(ctx, &runtime.StopContainerRequest{
ContainerId: cn,
Timeout: 3,
})
assert.Error(t, err)
t.Log("The container should still be running even after 5 seconds")
assert.NoError(t, Consistently(func() (bool, error) {
s, err := runtimeService.ContainerStatus(cn)
if err != nil {
return false, err
}
return s.GetState() == runtime.ContainerState_CONTAINER_RUNNING, nil
}, 100*time.Millisecond, 5*time.Second))
t.Log("Stop the container with 1s timeout, without shorter context timeout")
assert.NoError(t, runtimeService.StopContainer(cn, 1))
t.Log("The container state should be exited")
s, err := runtimeService.ContainerStatus(cn)
require.NoError(t, err)
assert.Equal(t, s.GetState(), runtime.ContainerState_CONTAINER_EXITED)
}

107
integration/container_update_resources_test.go Normal file
View File

@ -0,0 +1,107 @@
// +build linux
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package integration
import (
"testing"
"github.com/containerd/cgroups"
runtimespec "github.com/opencontainers/runtime-spec/specs-go"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"golang.org/x/net/context"
runtime "k8s.io/cri-api/pkg/apis/runtime/v1alpha2"
)
func checkMemoryLimit(t *testing.T, spec *runtimespec.Spec, memLimit int64) {
require.NotNil(t, spec)
require.NotNil(t, spec.Linux)
require.NotNil(t, spec.Linux.Resources)
require.NotNil(t, spec.Linux.Resources.Memory)
require.NotNil(t, spec.Linux.Resources.Memory.Limit)
assert.Equal(t, memLimit, *spec.Linux.Resources.Memory.Limit)
}
func TestUpdateContainerResources(t *testing.T) {
t.Log("Create a sandbox")
sbConfig := PodSandboxConfig("sandbox", "update-container-resources")
sb, err := runtimeService.RunPodSandbox(sbConfig, *runtimeHandler)
require.NoError(t, err)
defer func() {
assert.NoError(t, runtimeService.StopPodSandbox(sb))
assert.NoError(t, runtimeService.RemovePodSandbox(sb))
}()
t.Log("Create a container with memory limit")
cnConfig := ContainerConfig(
"container",
pauseImage,
WithResources(&runtime.LinuxContainerResources{
MemoryLimitInBytes: 200 * 1024 * 1024,
}),
)
cn, err := runtimeService.CreateContainer(sb, cnConfig, sbConfig)
require.NoError(t, err)
t.Log("Check memory limit in container OCI spec")
container, err := containerdClient.LoadContainer(context.Background(), cn)
require.NoError(t, err)
spec, err := container.Spec(context.Background())
require.NoError(t, err)
checkMemoryLimit(t, spec, 200*1024*1024)
t.Log("Update container memory limit after created")
err = runtimeService.UpdateContainerResources(cn, &runtime.LinuxContainerResources{
MemoryLimitInBytes: 400 * 1024 * 1024,
})
require.NoError(t, err)
t.Log("Check memory limit in container OCI spec")
spec, err = container.Spec(context.Background())
require.NoError(t, err)
checkMemoryLimit(t, spec, 400*1024*1024)
t.Log("Start the container")
require.NoError(t, runtimeService.StartContainer(cn))
task, err := container.Task(context.Background(), nil)
require.NoError(t, err)
t.Log("Check memory limit in cgroup")
cgroup, err := cgroups.Load(cgroups.V1, cgroups.PidPath(int(task.Pid())))
require.NoError(t, err)
stat, err := cgroup.Stat(cgroups.IgnoreNotExist)
require.NoError(t, err)
assert.Equal(t, uint64(400*1024*1024), stat.Memory.Usage.Limit)
t.Log("Update container memory limit after started")
err = runtimeService.UpdateContainerResources(cn, &runtime.LinuxContainerResources{
MemoryLimitInBytes: 800 * 1024 * 1024,
})
require.NoError(t, err)
t.Log("Check memory limit in container OCI spec")
spec, err = container.Spec(context.Background())
require.NoError(t, err)
checkMemoryLimit(t, spec, 800*1024*1024)
t.Log("Check memory limit in cgroup")
stat, err = cgroup.Stat(cgroups.IgnoreNotExist)
require.NoError(t, err)
assert.Equal(t, uint64(800*1024*1024), stat.Memory.Usage.Limit)
}

77
integration/container_without_image_ref_test.go Normal file
View File

@ -0,0 +1,77 @@
// +build linux
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package integration
import (
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
runtime "k8s.io/cri-api/pkg/apis/runtime/v1alpha2"
)
// Test container lifecycle can work without image references.
func TestContainerLifecycleWithoutImageRef(t *testing.T) {
t.Log("Create a sandbox")
sbConfig := PodSandboxConfig("sandbox", "container-lifecycle-without-image-ref")
sb, err := runtimeService.RunPodSandbox(sbConfig, *runtimeHandler)
require.NoError(t, err)
defer func() {
assert.NoError(t, runtimeService.StopPodSandbox(sb))
assert.NoError(t, runtimeService.RemovePodSandbox(sb))
}()
const (
testImage = "busybox"
containerName = "test-container"
)
t.Log("Pull test image")
img, err := imageService.PullImage(&runtime.ImageSpec{Image: testImage}, nil, sbConfig)
require.NoError(t, err)
defer func() {
assert.NoError(t, imageService.RemoveImage(&runtime.ImageSpec{Image: img}))
}()
t.Log("Create test container")
cnConfig := ContainerConfig(
containerName,
testImage,
WithCommand("sleep", "1000"),
)
cn, err := runtimeService.CreateContainer(sb, cnConfig, sbConfig)
require.NoError(t, err)
require.NoError(t, runtimeService.StartContainer(cn))
t.Log("Remove test image")
assert.NoError(t, imageService.RemoveImage(&runtime.ImageSpec{Image: img}))
t.Log("Container status should be running")
status, err := runtimeService.ContainerStatus(cn)
require.NoError(t, err)
assert.Equal(t, status.GetState(), runtime.ContainerState_CONTAINER_RUNNING)
t.Logf("Stop container")
err = runtimeService.StopContainer(cn, 1)
assert.NoError(t, err)
t.Log("Container status should be exited")
status, err = runtimeService.ContainerStatus(cn)
require.NoError(t, err)
assert.Equal(t, status.GetState(), runtime.ContainerState_CONTAINER_EXITED)
}

213
integration/containerd_image_test.go Normal file
View File

@ -0,0 +1,213 @@
// +build linux
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package integration
import (
"testing"
"time"
"golang.org/x/net/context"
"github.com/containerd/containerd"
"github.com/containerd/containerd/errdefs"
"github.com/containerd/containerd/namespaces"
"github.com/pkg/errors"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
runtime "k8s.io/cri-api/pkg/apis/runtime/v1alpha2"
)
// Test to test the CRI plugin should see image pulled into containerd directly.
func TestContainerdImage(t *testing.T) {
const testImage = "docker.io/library/busybox:latest"
ctx := context.Background()
t.Logf("make sure the test image doesn't exist in the cri plugin")
i, err := imageService.ImageStatus(&runtime.ImageSpec{Image: testImage})
require.NoError(t, err)
if i != nil {
require.NoError(t, imageService.RemoveImage(&runtime.ImageSpec{Image: testImage}))
}
t.Logf("pull the image into containerd")
_, err = containerdClient.Pull(ctx, testImage, containerd.WithPullUnpack)
assert.NoError(t, err)
defer func() {
// Make sure the image is cleaned up in any case.
if err := containerdClient.ImageService().Delete(ctx, testImage); err != nil {
assert.True(t, errdefs.IsNotFound(err), err)
}
assert.NoError(t, imageService.RemoveImage(&runtime.ImageSpec{Image: testImage}))
}()
t.Logf("the image should be seen by the cri plugin")
var id string
checkImage := func() (bool, error) {
img, err := imageService.ImageStatus(&runtime.ImageSpec{Image: testImage})
if err != nil {
return false, err
}
if img == nil {
t.Logf("Image %q not show up in the cri plugin yet", testImage)
return false, nil
}
id = img.Id
img, err = imageService.ImageStatus(&runtime.ImageSpec{Image: id})
if err != nil {
return false, err
}
if img == nil {
// We always generate image id as a reference first, it must
// be ready here.
return false, errors.New("can't reference image by id")
}
if len(img.RepoTags) != 1 {
// RepoTags must have been populated correctly.
return false, errors.Errorf("unexpected repotags: %+v", img.RepoTags)
}
if img.RepoTags[0] != testImage {
return false, errors.Errorf("unexpected repotag %q", img.RepoTags[0])
}
return true, nil
}
require.NoError(t, Eventually(checkImage, 100*time.Millisecond, 10*time.Second))
require.NoError(t, Consistently(checkImage, 100*time.Millisecond, time.Second))
defer func() {
t.Logf("image should still be seen by id if only tag get deleted")
if err := containerdClient.ImageService().Delete(ctx, testImage); err != nil {
assert.True(t, errdefs.IsNotFound(err), err)
}
assert.NoError(t, Consistently(func() (bool, error) {
img, err := imageService.ImageStatus(&runtime.ImageSpec{Image: id})
if err != nil {
return false, err
}
return img != nil, nil
}, 100*time.Millisecond, time.Second))
t.Logf("image should be removed from the cri plugin if all references get deleted")
if err := containerdClient.ImageService().Delete(ctx, id); err != nil {
assert.True(t, errdefs.IsNotFound(err), err)
}
assert.NoError(t, Eventually(func() (bool, error) {
img, err := imageService.ImageStatus(&runtime.ImageSpec{Image: id})
if err != nil {
return false, err
}
return img == nil, nil
}, 100*time.Millisecond, 10*time.Second))
}()
t.Logf("the image should be marked as managed")
imgByRef, err := containerdClient.GetImage(ctx, testImage)
assert.NoError(t, err)
assert.Equal(t, imgByRef.Labels()["io.cri-containerd.image"], "managed")
t.Logf("the image id should be created and managed")
imgByID, err := containerdClient.GetImage(ctx, id)
assert.NoError(t, err)
assert.Equal(t, imgByID.Labels()["io.cri-containerd.image"], "managed")
t.Logf("should be able to start container with the image")
sbConfig := PodSandboxConfig("sandbox", "containerd-image")
sb, err := runtimeService.RunPodSandbox(sbConfig, *runtimeHandler)
require.NoError(t, err)
defer func() {
assert.NoError(t, runtimeService.StopPodSandbox(sb))
assert.NoError(t, runtimeService.RemovePodSandbox(sb))
}()
cnConfig := ContainerConfig(
"test-container",
id,
WithCommand("top"),
)
cn, err := runtimeService.CreateContainer(sb, cnConfig, sbConfig)
require.NoError(t, err)
require.NoError(t, runtimeService.StartContainer(cn))
checkContainer := func() (bool, error) {
s, err := runtimeService.ContainerStatus(cn)
if err != nil {
return false, err
}
return s.GetState() == runtime.ContainerState_CONTAINER_RUNNING, nil
}
require.NoError(t, Eventually(checkContainer, 100*time.Millisecond, 10*time.Second))
require.NoError(t, Consistently(checkContainer, 100*time.Millisecond, time.Second))
}
// Test image managed by CRI plugin shouldn't be affected by images in other namespaces.
func TestContainerdImageInOtherNamespaces(t *testing.T) {
const testImage = "docker.io/library/busybox:latest"
ctx := context.Background()
t.Logf("make sure the test image doesn't exist in the cri plugin")
i, err := imageService.ImageStatus(&runtime.ImageSpec{Image: testImage})
require.NoError(t, err)
if i != nil {
require.NoError(t, imageService.RemoveImage(&runtime.ImageSpec{Image: testImage}))
}
t.Logf("pull the image into test namespace")
namespacedCtx := namespaces.WithNamespace(ctx, "test")
_, err = containerdClient.Pull(namespacedCtx, testImage, containerd.WithPullUnpack)
assert.NoError(t, err)
defer func() {
// Make sure the image is cleaned up in any case.
if err := containerdClient.ImageService().Delete(namespacedCtx, testImage); err != nil {
assert.True(t, errdefs.IsNotFound(err), err)
}
assert.NoError(t, imageService.RemoveImage(&runtime.ImageSpec{Image: testImage}))
}()
t.Logf("cri plugin should not see the image")
checkImage := func() (bool, error) {
img, err := imageService.ImageStatus(&runtime.ImageSpec{Image: testImage})
if err != nil {
return false, err
}
return img == nil, nil
}
require.NoError(t, Consistently(checkImage, 100*time.Millisecond, time.Second))
sbConfig := PodSandboxConfig("sandbox", "test")
t.Logf("pull the image into cri plugin")
id, err := imageService.PullImage(&runtime.ImageSpec{Image: testImage}, nil, sbConfig)
require.NoError(t, err)
defer func() {
assert.NoError(t, imageService.RemoveImage(&runtime.ImageSpec{Image: id}))
}()
t.Logf("cri plugin should see the image now")
img, err := imageService.ImageStatus(&runtime.ImageSpec{Image: testImage})
require.NoError(t, err)
assert.NotNil(t, img)
t.Logf("remove the image from test namespace")
require.NoError(t, containerdClient.ImageService().Delete(namespacedCtx, testImage))
t.Logf("cri plugin should still see the image")
checkImage = func() (bool, error) {
img, err := imageService.ImageStatus(&runtime.ImageSpec{Image: testImage})
if err != nil {
return false, err
}
return img != nil, nil
}
assert.NoError(t, Consistently(checkImage, 100*time.Millisecond, time.Second))
}

53
integration/duplicate_name_test.go Normal file
View File

@ -0,0 +1,53 @@
// +build linux
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package integration
import (
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestDuplicateName(t *testing.T) {
t.Logf("Create a sandbox")
sbConfig := PodSandboxConfig("sandbox", "duplicate-name")
sb, err := runtimeService.RunPodSandbox(sbConfig, *runtimeHandler)
require.NoError(t, err)
defer func() {
assert.NoError(t, runtimeService.StopPodSandbox(sb))
assert.NoError(t, runtimeService.RemovePodSandbox(sb))
}()
t.Logf("Create the sandbox again should fail")
_, err = runtimeService.RunPodSandbox(sbConfig, *runtimeHandler)
require.Error(t, err)
t.Logf("Create a container")
cnConfig := ContainerConfig(
"container",
pauseImage,
)
_, err = runtimeService.CreateContainer(sb, cnConfig, sbConfig)
require.NoError(t, err)
t.Logf("Create the container again should fail")
_, err = runtimeService.CreateContainer(sb, cnConfig, sbConfig)
require.Error(t, err)
}

101
integration/image_load_test.go Normal file
View File

@ -0,0 +1,101 @@
// +build linux
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package integration
import (
"io/ioutil"
"os"
"os/exec"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
runtime "k8s.io/cri-api/pkg/apis/runtime/v1alpha2"
)
// Test to load an image from tarball.
func TestImageLoad(t *testing.T) {
testImage := "busybox:latest"
loadedImage := "docker.io/library/" + testImage
_, err := exec.LookPath("docker")
if err != nil {
t.Skipf("Docker is not available: %v", err)
}
t.Logf("docker save image into tarball")
output, err := exec.Command("docker", "pull", testImage).CombinedOutput()
require.NoError(t, err, "output: %q", output)
tarF, err := ioutil.TempFile("", "image-load")
tar := tarF.Name()
require.NoError(t, err)
defer func() {
assert.NoError(t, os.RemoveAll(tar))
}()
output, err = exec.Command("docker", "save", testImage, "-o", tar).CombinedOutput()
require.NoError(t, err, "output: %q", output)
t.Logf("make sure no such image in cri")
img, err := imageService.ImageStatus(&runtime.ImageSpec{Image: testImage})
require.NoError(t, err)
if img != nil {
require.NoError(t, imageService.RemoveImage(&runtime.ImageSpec{Image: testImage}))
}
t.Logf("load image in cri")
ctr, err := exec.LookPath("ctr")
require.NoError(t, err, "ctr should be installed, make sure you've run `make install.deps`")
output, err = exec.Command(ctr, "-address="+containerdEndpoint,
"-n=k8s.io", "images", "import", tar).CombinedOutput()
require.NoError(t, err, "output: %q", output)
t.Logf("make sure image is loaded")
// Use Eventually because the cri plugin needs a short period of time
// to pick up images imported into containerd directly.
require.NoError(t, Eventually(func() (bool, error) {
img, err = imageService.ImageStatus(&runtime.ImageSpec{Image: testImage})
if err != nil {
return false, err
}
return img != nil, nil
}, 100*time.Millisecond, 10*time.Second))
require.Equal(t, []string{loadedImage}, img.RepoTags)
t.Logf("create a container with the loaded image")
sbConfig := PodSandboxConfig("sandbox", Randomize("image-load"))
sb, err := runtimeService.RunPodSandbox(sbConfig, *runtimeHandler)
require.NoError(t, err)
defer func() {
assert.NoError(t, runtimeService.StopPodSandbox(sb))
assert.NoError(t, runtimeService.RemovePodSandbox(sb))
}()
containerConfig := ContainerConfig(
"container",
testImage,
WithCommand("tail", "-f", "/dev/null"),
)
// Rely on sandbox clean to do container cleanup.
cn, err := runtimeService.CreateContainer(sb, containerConfig, sbConfig)
require.NoError(t, err)
require.NoError(t, runtimeService.StartContainer(cn))
t.Logf("make sure container is running")
status, err := runtimeService.ContainerStatus(cn)
require.NoError(t, err)
require.Equal(t, runtime.ContainerState_CONTAINER_RUNNING, status.State)
}

78
integration/imagefs_info_test.go Normal file
View File

@ -0,0 +1,78 @@
// +build linux
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package integration
import (
"os"
"testing"
"time"
"github.com/pkg/errors"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
runtime "k8s.io/cri-api/pkg/apis/runtime/v1alpha2"
)
func TestImageFSInfo(t *testing.T) {
config := PodSandboxConfig("running-pod", "imagefs")
t.Logf("Pull an image to make sure image fs is not empty")
img, err := imageService.PullImage(&runtime.ImageSpec{Image: "busybox"}, nil, config)
require.NoError(t, err)
defer func() {
err := imageService.RemoveImage(&runtime.ImageSpec{Image: img})
assert.NoError(t, err)
}()
t.Logf("Create a sandbox to make sure there is an active snapshot")
sb, err := runtimeService.RunPodSandbox(config, *runtimeHandler)
require.NoError(t, err)
defer func() {
assert.NoError(t, runtimeService.StopPodSandbox(sb))
assert.NoError(t, runtimeService.RemovePodSandbox(sb))
}()
// It takes time to populate imagefs stats. Use eventually
// to check for a period of time.
t.Logf("Check imagefs info")
var info *runtime.FilesystemUsage
require.NoError(t, Eventually(func() (bool, error) {
stats, err := imageService.ImageFsInfo()
if err != nil {
return false, err
}
if len(stats) == 0 {
return false, nil
}
if len(stats) >= 2 {
return false, errors.Errorf("unexpected stats length: %d", len(stats))
}
info = stats[0]
if info.GetTimestamp() != 0 &&
info.GetUsedBytes().GetValue() != 0 &&
info.GetInodesUsed().GetValue() != 0 &&
info.GetFsId().GetMountpoint() != "" {
return true, nil
}
return false, nil
}, time.Second, 30*time.Second))
t.Logf("Image filesystem mountpath should exist")
_, err = os.Stat(info.GetFsId().GetMountpoint())
assert.NoError(t, err)
}

17
integration/images/volume-copy-up/Dockerfile Normal file
View File

@ -0,0 +1,17 @@
# Copyright The containerd Authors.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
FROM busybox
RUN sh -c "mkdir /test_dir; echo test_content > /test_dir/test_file"
VOLUME "/test_dir"

27
integration/images/volume-copy-up/Makefile Normal file
View File

@ -0,0 +1,27 @@
# Copyright The containerd Authors.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
all: build
PROJ=gcr.io/k8s-cri-containerd
VERSION=1.0
IMAGE=$(PROJ)/volume-copy-up:$(VERSION)
build:
docker build -t $(IMAGE) .
push:
gcloud docker -- push $(IMAGE)
.PHONY: build push

18
integration/images/volume-ownership/Dockerfile Normal file
View File

@ -0,0 +1,18 @@
# Copyright The containerd Authors.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
FROM busybox
RUN mkdir -p /test_dir && \
chown -R nobody:nogroup /test_dir
VOLUME /test_dir

27
integration/images/volume-ownership/Makefile Normal file
View File

@ -0,0 +1,27 @@
# Copyright The containerd Authors.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
all: build
PROJ=gcr.io/k8s-cri-containerd
VERSION=1.0
IMAGE=$(PROJ)/volume-ownership:$(VERSION)
build:
docker build -t $(IMAGE) .
push:
gcloud docker -- push $(IMAGE)
.PHONY: build push

418
integration/main_test.go Normal file
View File

@ -0,0 +1,418 @@
// +build linux
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package integration
import (
"context"
"encoding/json"
"flag"
"fmt"
"os"
"os/exec"
"strconv"
"strings"
"testing"
"time"
"github.com/containerd/containerd"
"github.com/pkg/errors"
"github.com/sirupsen/logrus"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"google.golang.org/grpc"
cri "k8s.io/cri-api/pkg/apis"
runtime "k8s.io/cri-api/pkg/apis/runtime/v1alpha2"
"github.com/containerd/cri/integration/remote"
dialer "github.com/containerd/cri/integration/util"
criconfig "github.com/containerd/cri/pkg/config"
"github.com/containerd/cri/pkg/constants"
"github.com/containerd/cri/pkg/server"
"github.com/containerd/cri/pkg/util"
)
const (
timeout = 1 * time.Minute
pauseImage = "k8s.gcr.io/pause:3.2" // This is the same with default sandbox image.
k8sNamespace = constants.K8sContainerdNamespace
)
var (
runtimeService cri.RuntimeService
imageService cri.ImageManagerService
containerdClient *containerd.Client
containerdEndpoint string
)
var criEndpoint = flag.String("cri-endpoint", "unix:///run/containerd/containerd.sock", "The endpoint of cri plugin.")
var criRoot = flag.String("cri-root", "/var/lib/containerd/io.containerd.grpc.v1.cri", "The root directory of cri plugin.")
var runtimeHandler = flag.String("runtime-handler", "", "The runtime handler to use in the test.")
var containerdBin = flag.String("containerd-bin", "containerd", "The containerd binary name. The name is used to restart containerd during test.")
func TestMain(m *testing.M) {
flag.Parse()
if err := ConnectDaemons(); err != nil {
logrus.WithError(err).Fatalf("Failed to connect daemons")
}
os.Exit(m.Run())
}
// ConnectDaemons connect cri plugin and containerd, and initialize the clients.
func ConnectDaemons() error {
var err error
runtimeService, err = remote.NewRuntimeService(*criEndpoint, timeout)
if err != nil {
return errors.Wrap(err, "failed to create runtime service")
}
imageService, err = remote.NewImageService(*criEndpoint, timeout)
if err != nil {
return errors.Wrap(err, "failed to create image service")
}
// Since CRI grpc client doesn't have `WithBlock` specified, we
// need to check whether it is actually connected.
// TODO(random-liu): Extend cri remote client to accept extra grpc options.
_, err = runtimeService.ListContainers(&runtime.ContainerFilter{})
if err != nil {
return errors.Wrap(err, "failed to list containers")
}
_, err = imageService.ListImages(&runtime.ImageFilter{})
if err != nil {
return errors.Wrap(err, "failed to list images")
}
// containerdEndpoint is the same with criEndpoint now
containerdEndpoint = strings.TrimPrefix(*criEndpoint, "unix://")
containerdClient, err = containerd.New(containerdEndpoint, containerd.WithDefaultNamespace(k8sNamespace))
if err != nil {
return errors.Wrap(err, "failed to connect containerd")
}
return nil
}
// Opts sets specific information in pod sandbox config.
type PodSandboxOpts func(*runtime.PodSandboxConfig)
// Set host network.
func WithHostNetwork(p *runtime.PodSandboxConfig) {
if p.Linux == nil {
p.Linux = &runtime.LinuxPodSandboxConfig{}
}
if p.Linux.SecurityContext == nil {
p.Linux.SecurityContext = &runtime.LinuxSandboxSecurityContext{}
}
if p.Linux.SecurityContext.NamespaceOptions == nil {
p.Linux.SecurityContext.NamespaceOptions = &runtime.NamespaceOption{}
}
p.Linux.SecurityContext.NamespaceOptions.Network = runtime.NamespaceMode_NODE
}
// Set host pid.
func WithHostPid(p *runtime.PodSandboxConfig) {
if p.Linux == nil {
p.Linux = &runtime.LinuxPodSandboxConfig{}
}
if p.Linux.SecurityContext == nil {
p.Linux.SecurityContext = &runtime.LinuxSandboxSecurityContext{}
}
if p.Linux.SecurityContext.NamespaceOptions == nil {
p.Linux.SecurityContext.NamespaceOptions = &runtime.NamespaceOption{}
}
p.Linux.SecurityContext.NamespaceOptions.Pid = runtime.NamespaceMode_NODE
}
// Set pod pid.
func WithPodPid(p *runtime.PodSandboxConfig) {
if p.Linux == nil {
p.Linux = &runtime.LinuxPodSandboxConfig{}
}
if p.Linux.SecurityContext == nil {
p.Linux.SecurityContext = &runtime.LinuxSandboxSecurityContext{}
}
if p.Linux.SecurityContext.NamespaceOptions == nil {
p.Linux.SecurityContext.NamespaceOptions = &runtime.NamespaceOption{}
}
p.Linux.SecurityContext.NamespaceOptions.Pid = runtime.NamespaceMode_POD
}
// Add pod log directory.
func WithPodLogDirectory(dir string) PodSandboxOpts {
return func(p *runtime.PodSandboxConfig) {
p.LogDirectory = dir
}
}
// Add pod hostname.
func WithPodHostname(hostname string) PodSandboxOpts {
return func(p *runtime.PodSandboxConfig) {
p.Hostname = hostname
}
}
// PodSandboxConfig generates a pod sandbox config for test.
func PodSandboxConfig(name, ns string, opts ...PodSandboxOpts) *runtime.PodSandboxConfig {
config := &runtime.PodSandboxConfig{
Metadata: &runtime.PodSandboxMetadata{
Name: name,
// Using random id as uuid is good enough for local
// integration test.
Uid: util.GenerateID(),
Namespace: Randomize(ns),
},
Linux: &runtime.LinuxPodSandboxConfig{},
}
for _, opt := range opts {
opt(config)
}
return config
}
// ContainerOpts to set any specific attribute like labels,
// annotations, metadata etc
type ContainerOpts func(*runtime.ContainerConfig)
func WithTestLabels() ContainerOpts {
return func(c *runtime.ContainerConfig) {
c.Labels = map[string]string{"key": "value"}
}
}
func WithTestAnnotations() ContainerOpts {
return func(c *runtime.ContainerConfig) {
c.Annotations = map[string]string{"a.b.c": "test"}
}
}
// Add container resource limits.
func WithResources(r *runtime.LinuxContainerResources) ContainerOpts {
return func(c *runtime.ContainerConfig) {
if c.Linux == nil {
c.Linux = &runtime.LinuxContainerConfig{}
}
c.Linux.Resources = r
}
}
// Add container command.
func WithCommand(cmd string, args ...string) ContainerOpts {
return func(c *runtime.ContainerConfig) {
c.Command = []string{cmd}
c.Args = args
}
}
// Add pid namespace mode.
func WithPidNamespace(mode runtime.NamespaceMode) ContainerOpts {
return func(c *runtime.ContainerConfig) {
if c.Linux == nil {
c.Linux = &runtime.LinuxContainerConfig{}
}
if c.Linux.SecurityContext == nil {
c.Linux.SecurityContext = &runtime.LinuxContainerSecurityContext{}
}
if c.Linux.SecurityContext.NamespaceOptions == nil {
c.Linux.SecurityContext.NamespaceOptions = &runtime.NamespaceOption{}
}
c.Linux.SecurityContext.NamespaceOptions.Pid = mode
}
}
// Add container log path.
func WithLogPath(path string) ContainerOpts {
return func(c *runtime.ContainerConfig) {
c.LogPath = path
}
}
// WithSupplementalGroups adds supplemental groups.
func WithSupplementalGroups(gids []int64) ContainerOpts {
return func(c *runtime.ContainerConfig) {
if c.Linux == nil {
c.Linux = &runtime.LinuxContainerConfig{}
}
if c.Linux.SecurityContext == nil {
c.Linux.SecurityContext = &runtime.LinuxContainerSecurityContext{}
}
c.Linux.SecurityContext.SupplementalGroups = gids
}
}
// ContainerConfig creates a container config given a name and image name
// and additional container config options
func ContainerConfig(name, image string, opts ...ContainerOpts) *runtime.ContainerConfig {
cConfig := &runtime.ContainerConfig{
Metadata: &runtime.ContainerMetadata{
Name: name,
},
Image: &runtime.ImageSpec{Image: image},
}
for _, opt := range opts {
opt(cConfig)
}
return cConfig
}
// CheckFunc is the function used to check a condition is true/false.
type CheckFunc func() (bool, error)
// Eventually waits for f to return true, it checks every period, and
// returns error if timeout exceeds. If f returns error, Eventually
// will return the same error immediately.
func Eventually(f CheckFunc, period, timeout time.Duration) error {
start := time.Now()
for {
done, err := f()
if done {
return nil
}
if err != nil {
return err
}
if time.Since(start) >= timeout {
return errors.New("timeout exceeded")
}
time.Sleep(period)
}
}
// Consistently makes sure that f consistently returns true without
// error before timeout exceeds. If f returns error, Consistently
// will return the same error immediately.
func Consistently(f CheckFunc, period, timeout time.Duration) error {
start := time.Now()
for {
ok, err := f()
if !ok {
return errors.New("get false")
}
if err != nil {
return err
}
if time.Since(start) >= timeout {
return nil
}
time.Sleep(period)
}
}
// Randomize adds uuid after a string.
func Randomize(str string) string {
return str + "-" + util.GenerateID()
}
// KillProcess kills the process by name. pkill is used.
func KillProcess(name string) error {
output, err := exec.Command("pkill", "-x", fmt.Sprintf("^%s$", name)).CombinedOutput()
if err != nil {
return errors.Errorf("failed to kill %q - error: %v, output: %q", name, err, output)
}
return nil
}
// KillPid kills the process by pid. kill is used.
func KillPid(pid int) error {
output, err := exec.Command("kill", strconv.Itoa(pid)).CombinedOutput()
if err != nil {
return errors.Errorf("failed to kill %d - error: %v, output: %q", pid, err, output)
}
return nil
}
// PidOf returns pid of a process by name.
func PidOf(name string) (int, error) {
b, err := exec.Command("pidof", name).CombinedOutput()
output := strings.TrimSpace(string(b))
if err != nil {
if len(output) != 0 {
return 0, errors.Errorf("failed to run pidof %q - error: %v, output: %q", name, err, output)
}
return 0, nil
}
return strconv.Atoi(output)
}
// RawRuntimeClient returns a raw grpc runtime service client.
func RawRuntimeClient() (runtime.RuntimeServiceClient, error) {
addr, dialer, err := dialer.GetAddressAndDialer(*criEndpoint)
if err != nil {
return nil, errors.Wrap(err, "failed to get dialer")
}
ctx, cancel := context.WithTimeout(context.Background(), timeout)
defer cancel()
conn, err := grpc.DialContext(ctx, addr, grpc.WithInsecure(), grpc.WithContextDialer(dialer))
if err != nil {
return nil, errors.Wrap(err, "failed to connect cri endpoint")
}
return runtime.NewRuntimeServiceClient(conn), nil
}
// CRIConfig gets current cri config from containerd.
func CRIConfig() (*criconfig.Config, error) {
client, err := RawRuntimeClient()
if err != nil {
return nil, errors.Wrap(err, "failed to get raw runtime client")
}
resp, err := client.Status(context.Background(), &runtime.StatusRequest{Verbose: true})
if err != nil {
return nil, errors.Wrap(err, "failed to get status")
}
config := &criconfig.Config{}
if err := json.Unmarshal([]byte(resp.Info["config"]), config); err != nil {
return nil, errors.Wrap(err, "failed to unmarshal config")
}
return config, nil
}
// SandboxInfo gets sandbox info.
func SandboxInfo(id string) (*runtime.PodSandboxStatus, *server.SandboxInfo, error) {
client, err := RawRuntimeClient()
if err != nil {
return nil, nil, errors.Wrap(err, "failed to get raw runtime client")
}
resp, err := client.PodSandboxStatus(context.Background(), &runtime.PodSandboxStatusRequest{
PodSandboxId: id,
Verbose: true,
})
if err != nil {
return nil, nil, errors.Wrap(err, "failed to get sandbox status")
}
status := resp.GetStatus()
var info server.SandboxInfo
if err := json.Unmarshal([]byte(resp.GetInfo()["info"]), &info); err != nil {
return nil, nil, errors.Wrap(err, "failed to unmarshal sandbox info")
}
return status, &info, nil
}
func RestartContainerd(t *testing.T) {
require.NoError(t, KillProcess(*containerdBin))
// Use assert so that the 3rd wait always runs, this makes sure
// containerd is running before this function returns.
assert.NoError(t, Eventually(func() (bool, error) {
pid, err := PidOf(*containerdBin)
if err != nil {
return false, err
}
return pid == 0, nil
}, time.Second, 30*time.Second), "wait for containerd to be killed")
require.NoError(t, Eventually(func() (bool, error) {
return ConnectDaemons() == nil, nil
}, time.Second, 30*time.Second), "wait for containerd to be restarted")
}

50
integration/no_metadata_test.go Normal file
View File

@ -0,0 +1,50 @@
// +build linux
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package integration
import (
"testing"
"github.com/stretchr/testify/require"
runtime "k8s.io/cri-api/pkg/apis/runtime/v1alpha2"
)
func TestRunPodSandboxWithoutMetadata(t *testing.T) {
sbConfig := &runtime.PodSandboxConfig{}
_, err := runtimeService.RunPodSandbox(sbConfig, *runtimeHandler)
require.Error(t, err)
_, err = runtimeService.Status()
require.NoError(t, err)
}
func TestCreateContainerWithoutMetadata(t *testing.T) {
sbConfig := PodSandboxConfig("sandbox", "container-create")
sb, err := runtimeService.RunPodSandbox(sbConfig, *runtimeHandler)
require.NoError(t, err)
defer func() {
// Make sure the sandbox is cleaned up in any case.
runtimeService.StopPodSandbox(sb)
runtimeService.RemovePodSandbox(sb)
}()
config := &runtime.ContainerConfig{}
_, err = runtimeService.CreateContainer(sb, config, sbConfig)
require.Error(t, err)
_, err = runtimeService.Status()
require.NoError(t, err)
}

107
integration/pod_dualstack_test.go Normal file
View File

@ -0,0 +1,107 @@
// +build linux
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package integration
import (
"io/ioutil"
"net"
"os"
"path/filepath"
"regexp"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
runtime "k8s.io/cri-api/pkg/apis/runtime/v1alpha2"
)
func TestPodDualStack(t *testing.T) {
testPodLogDir, err := ioutil.TempDir("/tmp", "dualstack")
require.NoError(t, err)
defer os.RemoveAll(testPodLogDir)
t.Log("Create a sandbox")
sbConfig := PodSandboxConfig("sandbox", "dualstack", WithPodLogDirectory(testPodLogDir))
sb, err := runtimeService.RunPodSandbox(sbConfig, *runtimeHandler)
require.NoError(t, err)
defer func() {
assert.NoError(t, runtimeService.StopPodSandbox(sb))
assert.NoError(t, runtimeService.RemovePodSandbox(sb))
}()
const (
testImage = "busybox"
containerName = "test-container"
)
t.Logf("Pull test image %q", testImage)
img, err := imageService.PullImage(&runtime.ImageSpec{Image: testImage}, nil, sbConfig)
require.NoError(t, err)
defer func() {
assert.NoError(t, imageService.RemoveImage(&runtime.ImageSpec{Image: img}))
}()
t.Log("Create a container to print env")
cnConfig := ContainerConfig(
containerName,
testImage,
WithCommand("ip", "address", "show", "dev", "eth0"),
WithLogPath(containerName),
)
cn, err := runtimeService.CreateContainer(sb, cnConfig, sbConfig)
require.NoError(t, err)
t.Log("Start the container")
require.NoError(t, runtimeService.StartContainer(cn))
t.Log("Wait for container to finish running")
require.NoError(t, Eventually(func() (bool, error) {
s, err := runtimeService.ContainerStatus(cn)
if err != nil {
return false, err
}
if s.GetState() == runtime.ContainerState_CONTAINER_EXITED {
return true, nil
}
return false, nil
}, time.Second, 30*time.Second))
content, err := ioutil.ReadFile(filepath.Join(testPodLogDir, containerName))
assert.NoError(t, err)
status, err := runtimeService.PodSandboxStatus(sb)
require.NoError(t, err)
ip := status.GetNetwork().GetIp()
additionalIps := status.GetNetwork().GetAdditionalIps()
ipv4Enabled, err := regexp.MatchString("inet .* scope global", string(content))
assert.NoError(t, err)
ipv6Enabled, err := regexp.MatchString("inet6 .* scope global", string(content))
assert.NoError(t, err)
if ipv4Enabled && ipv6Enabled {
t.Log("Dualstack should be enabled")
require.Len(t, additionalIps, 1)
assert.NotNil(t, net.ParseIP(ip).To4())
assert.Nil(t, net.ParseIP(additionalIps[0].GetIp()).To4())
} else {
t.Log("Dualstack should not be enabled")
assert.Len(t, additionalIps, 0)
assert.NotEmpty(t, ip)
}
}

132
integration/pod_hostname_test.go Normal file
View File

@ -0,0 +1,132 @@
// +build linux
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package integration
import (
"io/ioutil"
"os"
"path/filepath"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
runtime "k8s.io/cri-api/pkg/apis/runtime/v1alpha2"
)
func TestPodHostname(t *testing.T) {
hostname, err := os.Hostname()
require.NoError(t, err)
for name, test := range map[string]struct {
opts []PodSandboxOpts
expectedHostname string
expectErr bool
}{
"regular pod with custom hostname": {
opts: []PodSandboxOpts{
WithPodHostname("test-hostname"),
},
expectedHostname: "test-hostname",
},
"host network pod without custom hostname": {
opts: []PodSandboxOpts{
WithHostNetwork,
},
expectedHostname: hostname,
},
"host network pod with custom hostname should fail": {
opts: []PodSandboxOpts{
WithHostNetwork,
WithPodHostname("test-hostname"),
},
expectErr: true,
},
} {
t.Run(name, func(t *testing.T) {
testPodLogDir, err := ioutil.TempDir("/tmp", "hostname")
require.NoError(t, err)
defer os.RemoveAll(testPodLogDir)
opts := append(test.opts, WithPodLogDirectory(testPodLogDir))
t.Log("Create a sandbox with hostname")
sbConfig := PodSandboxConfig("sandbox", "hostname", opts...)
sb, err := runtimeService.RunPodSandbox(sbConfig, *runtimeHandler)
if err != nil {
if !test.expectErr {
t.Fatalf("Unexpected RunPodSandbox error: %v", err)
}
return
}
// Make sure the sandbox is cleaned up.
defer func() {
assert.NoError(t, runtimeService.StopPodSandbox(sb))
assert.NoError(t, runtimeService.RemovePodSandbox(sb))
}()
if test.expectErr {
t.Fatalf("Expected RunPodSandbox to return error")
}
const (
testImage = "busybox"
containerName = "test-container"
)
t.Logf("Pull test image %q", testImage)
img, err := imageService.PullImage(&runtime.ImageSpec{Image: testImage}, nil, sbConfig)
require.NoError(t, err)
defer func() {
assert.NoError(t, imageService.RemoveImage(&runtime.ImageSpec{Image: img}))
}()
t.Log("Create a container to print env")
cnConfig := ContainerConfig(
containerName,
testImage,
WithCommand("sh", "-c",
"echo -n /etc/hostname= && cat /etc/hostname && env"),
WithLogPath(containerName),
)
cn, err := runtimeService.CreateContainer(sb, cnConfig, sbConfig)
require.NoError(t, err)
t.Log("Start the container")
require.NoError(t, runtimeService.StartContainer(cn))
t.Log("Wait for container to finish running")
require.NoError(t, Eventually(func() (bool, error) {
s, err := runtimeService.ContainerStatus(cn)
if err != nil {
return false, err
}
if s.GetState() == runtime.ContainerState_CONTAINER_EXITED {
return true, nil
}
return false, nil
}, time.Second, 30*time.Second))
content, err := ioutil.ReadFile(filepath.Join(testPodLogDir, containerName))
assert.NoError(t, err)
t.Log("Search hostname env in container log")
assert.Contains(t, string(content), "HOSTNAME="+test.expectedHostname)
t.Log("Search /etc/hostname content in container log")
assert.Contains(t, string(content), "/etc/hostname="+test.expectedHostname)
})
}
}

35
integration/remote/doc.go Normal file
View File

@ -0,0 +1,35 @@
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
/*
Copyright 2016 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// Package remote contains gRPC implementation of internalapi.RuntimeService
// and internalapi.ImageManagerService.
package remote

172
integration/remote/remote_image.go Normal file
View File

@ -0,0 +1,172 @@
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
/*
Copyright 2016 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package remote
import (
"context"
"errors"
"fmt"
"time"
"google.golang.org/grpc"
"k8s.io/klog/v2"
internalapi "k8s.io/cri-api/pkg/apis"
runtimeapi "k8s.io/cri-api/pkg/apis/runtime/v1alpha2"
"github.com/containerd/cri/integration/remote/util"
)
// ImageService is a gRPC implementation of internalapi.ImageManagerService.
type ImageService struct {
timeout time.Duration
imageClient runtimeapi.ImageServiceClient
}
// NewImageService creates a new internalapi.ImageManagerService.
func NewImageService(endpoint string, connectionTimeout time.Duration) (internalapi.ImageManagerService, error) {
klog.V(3).Infof("Connecting to image service %s", endpoint)
addr, dialer, err := util.GetAddressAndDialer(endpoint)
if err != nil {
return nil, err
}
ctx, cancel := context.WithTimeout(context.Background(), connectionTimeout)
defer cancel()
conn, err := grpc.DialContext(ctx, addr, grpc.WithInsecure(), grpc.WithContextDialer(dialer), grpc.WithDefaultCallOptions(grpc.MaxCallRecvMsgSize(maxMsgSize)))
if err != nil {
klog.Errorf("Connect remote image service %s failed: %v", addr, err)
return nil, err
}
return &ImageService{
timeout: connectionTimeout,
imageClient: runtimeapi.NewImageServiceClient(conn),
}, nil
}
// ListImages lists available images.
func (r *ImageService) ListImages(filter *runtimeapi.ImageFilter) ([]*runtimeapi.Image, error) {
ctx, cancel := getContextWithTimeout(r.timeout)
defer cancel()
resp, err := r.imageClient.ListImages(ctx, &runtimeapi.ListImagesRequest{
Filter: filter,
})
if err != nil {
klog.Errorf("ListImages with filter %+v from image service failed: %v", filter, err)
return nil, err
}
return resp.Images, nil
}
// ImageStatus returns the status of the image.
func (r *ImageService) ImageStatus(image *runtimeapi.ImageSpec) (*runtimeapi.Image, error) {
ctx, cancel := getContextWithTimeout(r.timeout)
defer cancel()
resp, err := r.imageClient.ImageStatus(ctx, &runtimeapi.ImageStatusRequest{
Image: image,
})
if err != nil {
klog.Errorf("ImageStatus %q from image service failed: %v", image.Image, err)
return nil, err
}
if resp.Image != nil {
if resp.Image.Id == "" || resp.Image.Size_ == 0 {
errorMessage := fmt.Sprintf("Id or size of image %q is not set", image.Image)
klog.Errorf("ImageStatus failed: %s", errorMessage)
return nil, errors.New(errorMessage)
}
}
return resp.Image, nil
}
// PullImage pulls an image with authentication config.
func (r *ImageService) PullImage(image *runtimeapi.ImageSpec, auth *runtimeapi.AuthConfig, podSandboxConfig *runtimeapi.PodSandboxConfig) (string, error) {
ctx, cancel := getContextWithCancel()
defer cancel()
resp, err := r.imageClient.PullImage(ctx, &runtimeapi.PullImageRequest{
Image: image,
Auth: auth,
SandboxConfig: podSandboxConfig,
})
if err != nil {
klog.Errorf("PullImage %q from image service failed: %v", image.Image, err)
return "", err
}
if resp.ImageRef == "" {
errorMessage := fmt.Sprintf("imageRef of image %q is not set", image.Image)
klog.Errorf("PullImage failed: %s", errorMessage)
return "", errors.New(errorMessage)
}
return resp.ImageRef, nil
}
// RemoveImage removes the image.
func (r *ImageService) RemoveImage(image *runtimeapi.ImageSpec) error {
ctx, cancel := getContextWithTimeout(r.timeout)
defer cancel()
_, err := r.imageClient.RemoveImage(ctx, &runtimeapi.RemoveImageRequest{
Image: image,
})
if err != nil {
klog.Errorf("RemoveImage %q from image service failed: %v", image.Image, err)
return err
}
return nil
}
// ImageFsInfo returns information of the filesystem that is used to store images.
func (r *ImageService) ImageFsInfo() ([]*runtimeapi.FilesystemUsage, error) {
// Do not set timeout, because `ImageFsInfo` takes time.
// TODO(random-liu): Should we assume runtime should cache the result, and set timeout here?
ctx, cancel := getContextWithCancel()
defer cancel()
resp, err := r.imageClient.ImageFsInfo(ctx, &runtimeapi.ImageFsInfoRequest{})
if err != nil {
klog.Errorf("ImageFsInfo from image service failed: %v", err)
return nil, err
}
return resp.GetImageFilesystems(), nil
}

586
integration/remote/remote_runtime.go Normal file
View File

@ -0,0 +1,586 @@
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
/*
Copyright 2016 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package remote
import (
"context"
"errors"
"fmt"
"strings"
"time"
"google.golang.org/grpc"
"k8s.io/klog/v2"
"k8s.io/component-base/logs/logreduction"
internalapi "k8s.io/cri-api/pkg/apis"
runtimeapi "k8s.io/cri-api/pkg/apis/runtime/v1alpha2"
utilexec "k8s.io/utils/exec"
"github.com/containerd/cri/integration/remote/util"
)
// RuntimeService is a gRPC implementation of internalapi.RuntimeService.
type RuntimeService struct {
timeout time.Duration
runtimeClient runtimeapi.RuntimeServiceClient
// Cache last per-container error message to reduce log spam
logReduction *logreduction.LogReduction
}
const (
// How frequently to report identical errors
identicalErrorDelay = 1 * time.Minute
)
// NewRuntimeService creates a new internalapi.RuntimeService.
func NewRuntimeService(endpoint string, connectionTimeout time.Duration) (internalapi.RuntimeService, error) {
klog.V(3).Infof("Connecting to runtime service %s", endpoint)
addr, dialer, err := util.GetAddressAndDialer(endpoint)
if err != nil {
return nil, err
}
ctx, cancel := context.WithTimeout(context.Background(), connectionTimeout)
defer cancel()
conn, err := grpc.DialContext(ctx, addr, grpc.WithInsecure(), grpc.WithContextDialer(dialer), grpc.WithDefaultCallOptions(grpc.MaxCallRecvMsgSize(maxMsgSize)))
if err != nil {
klog.Errorf("Connect remote runtime %s failed: %v", addr, err)
return nil, err
}
return &RuntimeService{
timeout: connectionTimeout,
runtimeClient: runtimeapi.NewRuntimeServiceClient(conn),
logReduction: logreduction.NewLogReduction(identicalErrorDelay),
}, nil
}
// Version returns the runtime name, runtime version and runtime API version.
func (r *RuntimeService) Version(apiVersion string) (*runtimeapi.VersionResponse, error) {
klog.V(10).Infof("[RuntimeService] Version (apiVersion=%v, timeout=%v)", apiVersion, r.timeout)
ctx, cancel := getContextWithTimeout(r.timeout)
defer cancel()
typedVersion, err := r.runtimeClient.Version(ctx, &runtimeapi.VersionRequest{
Version: apiVersion,
})
if err != nil {
klog.Errorf("Version from runtime service failed: %v", err)
return nil, err
}
klog.V(10).Infof("[RuntimeService] Version Response (typedVersion=%v)", typedVersion)
if typedVersion.Version == "" || typedVersion.RuntimeName == "" || typedVersion.RuntimeApiVersion == "" || typedVersion.RuntimeVersion == "" {
return nil, fmt.Errorf("not all fields are set in VersionResponse (%q)", *typedVersion)
}
return typedVersion, err
}
// RunPodSandbox creates and starts a pod-level sandbox. Runtimes should ensure
// the sandbox is in ready state.
func (r *RuntimeService) RunPodSandbox(config *runtimeapi.PodSandboxConfig, runtimeHandler string) (string, error) {
// Use 2 times longer timeout for sandbox operation (4 mins by default)
// TODO: Make the pod sandbox timeout configurable.
timeout := r.timeout * 2
klog.V(10).Infof("[RuntimeService] RunPodSandbox (config=%v, runtimeHandler=%v, timeout=%v)", config, runtimeHandler, timeout)
ctx, cancel := getContextWithTimeout(timeout)
defer cancel()
resp, err := r.runtimeClient.RunPodSandbox(ctx, &runtimeapi.RunPodSandboxRequest{
Config: config,
RuntimeHandler: runtimeHandler,
})
if err != nil {
klog.Errorf("RunPodSandbox from runtime service failed: %v", err)
return "", err
}
if resp.PodSandboxId == "" {
errorMessage := fmt.Sprintf("PodSandboxId is not set for sandbox %q", config.GetMetadata())
klog.Errorf("RunPodSandbox failed: %s", errorMessage)
return "", errors.New(errorMessage)
}
klog.V(10).Infof("[RuntimeService] RunPodSandbox Response (PodSandboxId=%v)", resp.PodSandboxId)
return resp.PodSandboxId, nil
}
// StopPodSandbox stops the sandbox. If there are any running containers in the
// sandbox, they should be forced to termination.
func (r *RuntimeService) StopPodSandbox(podSandBoxID string) error {
klog.V(10).Infof("[RuntimeService] StopPodSandbox (podSandboxID=%v, timeout=%v)", podSandBoxID, r.timeout)
ctx, cancel := getContextWithTimeout(r.timeout)
defer cancel()
_, err := r.runtimeClient.StopPodSandbox(ctx, &runtimeapi.StopPodSandboxRequest{
PodSandboxId: podSandBoxID,
})
if err != nil {
klog.Errorf("StopPodSandbox %q from runtime service failed: %v", podSandBoxID, err)
return err
}
klog.V(10).Infof("[RuntimeService] StopPodSandbox Response (podSandboxID=%v)", podSandBoxID)
return nil
}
// RemovePodSandbox removes the sandbox. If there are any containers in the
// sandbox, they should be forcibly removed.
func (r *RuntimeService) RemovePodSandbox(podSandBoxID string) error {
klog.V(10).Infof("[RuntimeService] RemovePodSandbox (podSandboxID=%v, timeout=%v)", podSandBoxID, r.timeout)
ctx, cancel := getContextWithTimeout(r.timeout)
defer cancel()
_, err := r.runtimeClient.RemovePodSandbox(ctx, &runtimeapi.RemovePodSandboxRequest{
PodSandboxId: podSandBoxID,
})
if err != nil {
klog.Errorf("RemovePodSandbox %q from runtime service failed: %v", podSandBoxID, err)
return err
}
klog.V(10).Infof("[RuntimeService] RemovePodSandbox Response (podSandboxID=%v)", podSandBoxID)
return nil
}
// PodSandboxStatus returns the status of the PodSandbox.
func (r *RuntimeService) PodSandboxStatus(podSandBoxID string) (*runtimeapi.PodSandboxStatus, error) {
klog.V(10).Infof("[RuntimeService] PodSandboxStatus (podSandboxID=%v, timeout=%v)", podSandBoxID, r.timeout)
ctx, cancel := getContextWithTimeout(r.timeout)
defer cancel()
resp, err := r.runtimeClient.PodSandboxStatus(ctx, &runtimeapi.PodSandboxStatusRequest{
PodSandboxId: podSandBoxID,
})
if err != nil {
return nil, err
}
klog.V(10).Infof("[RuntimeService] PodSandboxStatus Response (podSandboxID=%v, status=%v)", podSandBoxID, resp.Status)
if resp.Status != nil {
if err := verifySandboxStatus(resp.Status); err != nil {
return nil, err
}
}
return resp.Status, nil
}
// ListPodSandbox returns a list of PodSandboxes.
func (r *RuntimeService) ListPodSandbox(filter *runtimeapi.PodSandboxFilter) ([]*runtimeapi.PodSandbox, error) {
klog.V(10).Infof("[RuntimeService] ListPodSandbox (filter=%v, timeout=%v)", filter, r.timeout)
ctx, cancel := getContextWithTimeout(r.timeout)
defer cancel()
resp, err := r.runtimeClient.ListPodSandbox(ctx, &runtimeapi.ListPodSandboxRequest{
Filter: filter,
})
if err != nil {
klog.Errorf("ListPodSandbox with filter %+v from runtime service failed: %v", filter, err)
return nil, err
}
klog.V(10).Infof("[RuntimeService] ListPodSandbox Response (filter=%v, items=%v)", filter, resp.Items)
return resp.Items, nil
}
// CreateContainer creates a new container in the specified PodSandbox.
func (r *RuntimeService) CreateContainer(podSandBoxID string, config *runtimeapi.ContainerConfig, sandboxConfig *runtimeapi.PodSandboxConfig) (string, error) {
klog.V(10).Infof("[RuntimeService] CreateContainer (podSandBoxID=%v, timeout=%v)", podSandBoxID, r.timeout)
ctx, cancel := getContextWithTimeout(r.timeout)
defer cancel()
resp, err := r.runtimeClient.CreateContainer(ctx, &runtimeapi.CreateContainerRequest{
PodSandboxId: podSandBoxID,
Config: config,
SandboxConfig: sandboxConfig,
})
if err != nil {
klog.Errorf("CreateContainer in sandbox %q from runtime service failed: %v", podSandBoxID, err)
return "", err
}
klog.V(10).Infof("[RuntimeService] CreateContainer (podSandBoxID=%v, ContainerId=%v)", podSandBoxID, resp.ContainerId)
if resp.ContainerId == "" {
errorMessage := fmt.Sprintf("ContainerId is not set for container %q", config.GetMetadata())
klog.Errorf("CreateContainer failed: %s", errorMessage)
return "", errors.New(errorMessage)
}
return resp.ContainerId, nil
}
// StartContainer starts the container.
func (r *RuntimeService) StartContainer(containerID string) error {
klog.V(10).Infof("[RuntimeService] StartContainer (containerID=%v, timeout=%v)", containerID, r.timeout)
ctx, cancel := getContextWithTimeout(r.timeout)
defer cancel()
_, err := r.runtimeClient.StartContainer(ctx, &runtimeapi.StartContainerRequest{
ContainerId: containerID,
})
if err != nil {
klog.Errorf("StartContainer %q from runtime service failed: %v", containerID, err)
return err
}
klog.V(10).Infof("[RuntimeService] StartContainer Response (containerID=%v)", containerID)
return nil
}
// StopContainer stops a running container with a grace period (i.e., timeout).
func (r *RuntimeService) StopContainer(containerID string, timeout int64) error {
klog.V(10).Infof("[RuntimeService] StopContainer (containerID=%v, timeout=%v)", containerID, timeout)
// Use timeout + default timeout (2 minutes) as timeout to leave extra time
// for SIGKILL container and request latency.
t := r.timeout + time.Duration(timeout)*time.Second
ctx, cancel := getContextWithTimeout(t)
defer cancel()
r.logReduction.ClearID(containerID)
_, err := r.runtimeClient.StopContainer(ctx, &runtimeapi.StopContainerRequest{
ContainerId: containerID,
Timeout: timeout,
})
if err != nil {
klog.Errorf("StopContainer %q from runtime service failed: %v", containerID, err)
return err
}
klog.V(10).Infof("[RuntimeService] StopContainer Response (containerID=%v)", containerID)
return nil
}
// RemoveContainer removes the container. If the container is running, the container
// should be forced to removal.
func (r *RuntimeService) RemoveContainer(containerID string) error {
klog.V(10).Infof("[RuntimeService] RemoveContainer (containerID=%v, timeout=%v)", containerID, r.timeout)
ctx, cancel := getContextWithTimeout(r.timeout)
defer cancel()
r.logReduction.ClearID(containerID)
_, err := r.runtimeClient.RemoveContainer(ctx, &runtimeapi.RemoveContainerRequest{
ContainerId: containerID,
})
if err != nil {
klog.Errorf("RemoveContainer %q from runtime service failed: %v", containerID, err)
return err
}
klog.V(10).Infof("[RuntimeService] RemoveContainer Response (containerID=%v)", containerID)
return nil
}
// ListContainers lists containers by filters.
func (r *RuntimeService) ListContainers(filter *runtimeapi.ContainerFilter) ([]*runtimeapi.Container, error) {
klog.V(10).Infof("[RuntimeService] ListContainers (filter=%v, timeout=%v)", filter, r.timeout)
ctx, cancel := getContextWithTimeout(r.timeout)
defer cancel()
resp, err := r.runtimeClient.ListContainers(ctx, &runtimeapi.ListContainersRequest{
Filter: filter,
})
if err != nil {
klog.Errorf("ListContainers with filter %+v from runtime service failed: %v", filter, err)
return nil, err
}
klog.V(10).Infof("[RuntimeService] ListContainers Response (filter=%v, containers=%v)", filter, resp.Containers)
return resp.Containers, nil
}
// ContainerStatus returns the container status.
func (r *RuntimeService) ContainerStatus(containerID string) (*runtimeapi.ContainerStatus, error) {
klog.V(10).Infof("[RuntimeService] ContainerStatus (containerID=%v, timeout=%v)", containerID, r.timeout)
ctx, cancel := getContextWithTimeout(r.timeout)
defer cancel()
resp, err := r.runtimeClient.ContainerStatus(ctx, &runtimeapi.ContainerStatusRequest{
ContainerId: containerID,
})
if err != nil {
// Don't spam the log with endless messages about the same failure.
if r.logReduction.ShouldMessageBePrinted(err.Error(), containerID) {
klog.Errorf("ContainerStatus %q from runtime service failed: %v", containerID, err)
}
return nil, err
}
r.logReduction.ClearID(containerID)
klog.V(10).Infof("[RuntimeService] ContainerStatus Response (containerID=%v, status=%v)", containerID, resp.Status)
if resp.Status != nil {
if err := verifyContainerStatus(resp.Status); err != nil {
klog.Errorf("ContainerStatus of %q failed: %v", containerID, err)
return nil, err
}
}
return resp.Status, nil
}
// UpdateContainerResources updates a containers resource config
func (r *RuntimeService) UpdateContainerResources(containerID string, resources *runtimeapi.LinuxContainerResources) error {
klog.V(10).Infof("[RuntimeService] UpdateContainerResources (containerID=%v, timeout=%v)", containerID, r.timeout)
ctx, cancel := getContextWithTimeout(r.timeout)
defer cancel()
_, err := r.runtimeClient.UpdateContainerResources(ctx, &runtimeapi.UpdateContainerResourcesRequest{
ContainerId: containerID,
Linux: resources,
})
if err != nil {
klog.Errorf("UpdateContainerResources %q from runtime service failed: %v", containerID, err)
return err
}
klog.V(10).Infof("[RuntimeService] UpdateContainerResources Response (containerID=%v)", containerID)
return nil
}
// ExecSync executes a command in the container, and returns the stdout output.
// If command exits with a non-zero exit code, an error is returned.
func (r *RuntimeService) ExecSync(containerID string, cmd []string, timeout time.Duration) (stdout []byte, stderr []byte, err error) {
klog.V(10).Infof("[RuntimeService] ExecSync (containerID=%v, timeout=%v)", containerID, timeout)
// Do not set timeout when timeout is 0.
var ctx context.Context
var cancel context.CancelFunc
if timeout != 0 {
// Use timeout + default timeout (2 minutes) as timeout to leave some time for
// the runtime to do cleanup.
ctx, cancel = getContextWithTimeout(r.timeout + timeout)
} else {
ctx, cancel = getContextWithCancel()
}
defer cancel()
timeoutSeconds := int64(timeout.Seconds())
req := &runtimeapi.ExecSyncRequest{
ContainerId: containerID,
Cmd: cmd,
Timeout: timeoutSeconds,
}
resp, err := r.runtimeClient.ExecSync(ctx, req)
if err != nil {
klog.Errorf("ExecSync %s '%s' from runtime service failed: %v", containerID, strings.Join(cmd, " "), err)
return nil, nil, err
}
klog.V(10).Infof("[RuntimeService] ExecSync Response (containerID=%v, ExitCode=%v)", containerID, resp.ExitCode)
err = nil
if resp.ExitCode != 0 {
err = utilexec.CodeExitError{
Err: fmt.Errorf("command '%s' exited with %d: %s", strings.Join(cmd, " "), resp.ExitCode, resp.Stderr),
Code: int(resp.ExitCode),
}
}
return resp.Stdout, resp.Stderr, err
}
// Exec prepares a streaming endpoint to execute a command in the container, and returns the address.
func (r *RuntimeService) Exec(req *runtimeapi.ExecRequest) (*runtimeapi.ExecResponse, error) {
klog.V(10).Infof("[RuntimeService] Exec (timeout=%v)", r.timeout)
ctx, cancel := getContextWithTimeout(r.timeout)
defer cancel()
resp, err := r.runtimeClient.Exec(ctx, req)
if err != nil {
klog.Errorf("Exec %s '%s' from runtime service failed: %v", req.ContainerId, strings.Join(req.Cmd, " "), err)
return nil, err
}
klog.V(10).Info("[RuntimeService] Exec Response")
if resp.Url == "" {
errorMessage := "URL is not set"
klog.Errorf("Exec failed: %s", errorMessage)
return nil, errors.New(errorMessage)
}
return resp, nil
}
// Attach prepares a streaming endpoint to attach to a running container, and returns the address.
func (r *RuntimeService) Attach(req *runtimeapi.AttachRequest) (*runtimeapi.AttachResponse, error) {
klog.V(10).Infof("[RuntimeService] Attach (containerId=%v, timeout=%v)", req.ContainerId, r.timeout)
ctx, cancel := getContextWithTimeout(r.timeout)
defer cancel()
resp, err := r.runtimeClient.Attach(ctx, req)
if err != nil {
klog.Errorf("Attach %s from runtime service failed: %v", req.ContainerId, err)
return nil, err
}
klog.V(10).Infof("[RuntimeService] Attach Response (containerId=%v)", req.ContainerId)
if resp.Url == "" {
errorMessage := "URL is not set"
klog.Errorf("Attach failed: %s", errorMessage)
return nil, errors.New(errorMessage)
}
return resp, nil
}
// PortForward prepares a streaming endpoint to forward ports from a PodSandbox, and returns the address.
func (r *RuntimeService) PortForward(req *runtimeapi.PortForwardRequest) (*runtimeapi.PortForwardResponse, error) {
klog.V(10).Infof("[RuntimeService] PortForward (podSandboxID=%v, port=%v, timeout=%v)", req.PodSandboxId, req.Port, r.timeout)
ctx, cancel := getContextWithTimeout(r.timeout)
defer cancel()
resp, err := r.runtimeClient.PortForward(ctx, req)
if err != nil {
klog.Errorf("PortForward %s from runtime service failed: %v", req.PodSandboxId, err)
return nil, err
}
klog.V(10).Infof("[RuntimeService] PortForward Response (podSandboxID=%v)", req.PodSandboxId)
if resp.Url == "" {
errorMessage := "URL is not set"
klog.Errorf("PortForward failed: %s", errorMessage)
return nil, errors.New(errorMessage)
}
return resp, nil
}
// UpdateRuntimeConfig updates the config of a runtime service. The only
// update payload currently supported is the pod CIDR assigned to a node,
// and the runtime service just proxies it down to the network plugin.
func (r *RuntimeService) UpdateRuntimeConfig(runtimeConfig *runtimeapi.RuntimeConfig) error {
klog.V(10).Infof("[RuntimeService] UpdateRuntimeConfig (runtimeConfig=%v, timeout=%v)", runtimeConfig, r.timeout)
ctx, cancel := getContextWithTimeout(r.timeout)
defer cancel()
// Response doesn't contain anything of interest. This translates to an
// Event notification to the network plugin, which can't fail, so we're
// really looking to surface destination unreachable.
_, err := r.runtimeClient.UpdateRuntimeConfig(ctx, &runtimeapi.UpdateRuntimeConfigRequest{
RuntimeConfig: runtimeConfig,
})
if err != nil {
return err
}
klog.V(10).Infof("[RuntimeService] UpdateRuntimeConfig Response (runtimeConfig=%v)", runtimeConfig)
return nil
}
// Status returns the status of the runtime.
func (r *RuntimeService) Status() (*runtimeapi.RuntimeStatus, error) {
klog.V(10).Infof("[RuntimeService] Status (timeout=%v)", r.timeout)
ctx, cancel := getContextWithTimeout(r.timeout)
defer cancel()
resp, err := r.runtimeClient.Status(ctx, &runtimeapi.StatusRequest{})
if err != nil {
klog.Errorf("Status from runtime service failed: %v", err)
return nil, err
}
klog.V(10).Infof("[RuntimeService] Status Response (status=%v)", resp.Status)
if resp.Status == nil || len(resp.Status.Conditions) < 2 {
errorMessage := "RuntimeReady or NetworkReady condition are not set"
klog.Errorf("Status failed: %s", errorMessage)
return nil, errors.New(errorMessage)
}
return resp.Status, nil
}
// ContainerStats returns the stats of the container.
func (r *RuntimeService) ContainerStats(containerID string) (*runtimeapi.ContainerStats, error) {
klog.V(10).Infof("[RuntimeService] ContainerStats (containerID=%v, timeout=%v)", containerID, r.timeout)
ctx, cancel := getContextWithTimeout(r.timeout)
defer cancel()
resp, err := r.runtimeClient.ContainerStats(ctx, &runtimeapi.ContainerStatsRequest{
ContainerId: containerID,
})
if err != nil {
if r.logReduction.ShouldMessageBePrinted(err.Error(), containerID) {
klog.Errorf("ContainerStats %q from runtime service failed: %v", containerID, err)
}
return nil, err
}
r.logReduction.ClearID(containerID)
klog.V(10).Infof("[RuntimeService] ContainerStats Response (containerID=%v, stats=%v)", containerID, resp.GetStats())
return resp.GetStats(), nil
}
func (r *RuntimeService) ListContainerStats(filter *runtimeapi.ContainerStatsFilter) ([]*runtimeapi.ContainerStats, error) {
klog.V(10).Infof("[RuntimeService] ListContainerStats (filter=%v)", filter)
// Do not set timeout, because writable layer stats collection takes time.
// TODO(random-liu): Should we assume runtime should cache the result, and set timeout here?
ctx, cancel := getContextWithCancel()
defer cancel()
resp, err := r.runtimeClient.ListContainerStats(ctx, &runtimeapi.ListContainerStatsRequest{
Filter: filter,
})
if err != nil {
klog.Errorf("ListContainerStats with filter %+v from runtime service failed: %v", filter, err)
return nil, err
}
klog.V(10).Infof("[RuntimeService] ListContainerStats Response (filter=%v, stats=%v)", filter, resp.GetStats())
return resp.GetStats(), nil
}
func (r *RuntimeService) ReopenContainerLog(containerID string) error {
klog.V(10).Infof("[RuntimeService] ReopenContainerLog (containerID=%v, timeout=%v)", containerID, r.timeout)
ctx, cancel := getContextWithTimeout(r.timeout)
defer cancel()
_, err := r.runtimeClient.ReopenContainerLog(ctx, &runtimeapi.ReopenContainerLogRequest{ContainerId: containerID})
if err != nil {
klog.Errorf("ReopenContainerLog %q from runtime service failed: %v", containerID, err)
return err
}
klog.V(10).Infof("[RuntimeService] ReopenContainerLog Response (containerID=%v)", containerID)
return nil
}

161
integration/remote/util/util_unix.go Normal file
View File

@ -0,0 +1,161 @@
// +build freebsd linux darwin
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
/*
Copyright 2017 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package util
import (
"context"
"fmt"
"io/ioutil"
"net"
"net/url"
"os"
"path/filepath"
"golang.org/x/sys/unix"
"k8s.io/klog/v2"
)
const (
// unixProtocol is the network protocol of unix socket.
unixProtocol = "unix"
)
// CreateListener creates a listener on the specified endpoint.
func CreateListener(endpoint string) (net.Listener, error) {
protocol, addr, err := parseEndpointWithFallbackProtocol(endpoint, unixProtocol)
if err != nil {
return nil, err
}
if protocol != unixProtocol {
return nil, fmt.Errorf("only support unix socket endpoint")
}
// Unlink to cleanup the previous socket file.
err = unix.Unlink(addr)
if err != nil && !os.IsNotExist(err) {
return nil, fmt.Errorf("failed to unlink socket file %q: %v", addr, err)
}
if err := os.MkdirAll(filepath.Dir(addr), 0750); err != nil {
return nil, fmt.Errorf("error creating socket directory %q: %v", filepath.Dir(addr), err)
}
// Create the socket on a tempfile and move it to the destination socket to handle improprer cleanup
file, err := ioutil.TempFile(filepath.Dir(addr), "")
if err != nil {
return nil, fmt.Errorf("failed to create temporary file: %v", err)
}
if err := os.Remove(file.Name()); err != nil {
return nil, fmt.Errorf("failed to remove temporary file: %v", err)
}
l, err := net.Listen(protocol, file.Name())
if err != nil {
return nil, err
}
if err = os.Rename(file.Name(), addr); err != nil {
return nil, fmt.Errorf("failed to move temporary file to addr %q: %v", addr, err)
}
return l, nil
}
// GetAddressAndDialer returns the address parsed from the given endpoint and a context dialer.
func GetAddressAndDialer(endpoint string) (string, func(ctx context.Context, addr string) (net.Conn, error), error) {
protocol, addr, err := parseEndpointWithFallbackProtocol(endpoint, unixProtocol)
if err != nil {
return "", nil, err
}
if protocol != unixProtocol {
return "", nil, fmt.Errorf("only support unix socket endpoint")
}
return addr, dial, nil
}
func dial(ctx context.Context, addr string) (net.Conn, error) {
return (&net.Dialer{}).DialContext(ctx, unixProtocol, addr)
}
func parseEndpointWithFallbackProtocol(endpoint string, fallbackProtocol string) (protocol string, addr string, err error) {
if protocol, addr, err = parseEndpoint(endpoint); err != nil && protocol == "" {
fallbackEndpoint := fallbackProtocol + "://" + endpoint
protocol, addr, err = parseEndpoint(fallbackEndpoint)
if err == nil {
klog.Warningf("Using %q as endpoint is deprecated, please consider using full url format %q.", endpoint, fallbackEndpoint)
}
}
return
}
func parseEndpoint(endpoint string) (string, string, error) {
u, err := url.Parse(endpoint)
if err != nil {
return "", "", err
}
switch u.Scheme {
case "tcp":
return "tcp", u.Host, nil
case "unix":
return "unix", u.Path, nil
case "":
return "", "", fmt.Errorf("using %q as endpoint is deprecated, please consider using full url format", endpoint)
default:
return u.Scheme, "", fmt.Errorf("protocol %q not supported", u.Scheme)
}
}
// IsUnixDomainSocket returns whether a given file is a AF_UNIX socket file
func IsUnixDomainSocket(filePath string) (bool, error) {
fi, err := os.Stat(filePath)
if err != nil {
return false, fmt.Errorf("stat file %s failed: %v", filePath, err)
}
if fi.Mode()&os.ModeSocket == 0 {
return false, nil
}
return true, nil
}
// NormalizePath is a no-op for Linux for now
func NormalizePath(path string) string {
return path
}

71
integration/remote/util/util_unsupported.go Normal file
View File

@ -0,0 +1,71 @@
// +build !freebsd,!linux,!windows,!darwin
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
/*
Copyright 2017 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package util
import (
"context"
"fmt"
"net"
"time"
)
// CreateListener creates a listener on the specified endpoint.
func CreateListener(endpoint string) (net.Listener, error) {
return nil, fmt.Errorf("CreateListener is unsupported in this build")
}
// GetAddressAndDialer returns the address parsed from the given endpoint and a context dialer.
func GetAddressAndDialer(endpoint string) (string, func(ctx context.Context, addr string) (net.Conn, error), error) {
return "", nil, fmt.Errorf("GetAddressAndDialer is unsupported in this build")
}
// LockAndCheckSubPath empty implementation
func LockAndCheckSubPath(volumePath, subPath string) ([]uintptr, error) {
return []uintptr{}, nil
}
// UnlockPath empty implementation
func UnlockPath(fileHandles []uintptr) {
}
// LocalEndpoint empty implementation
func LocalEndpoint(path, file string) (string, error) {
return "", fmt.Errorf("LocalEndpoints are unsupported in this build")
}
// GetBootTime empty implementation
func GetBootTime() (time.Time, error) {
return time.Time{}, fmt.Errorf("GetBootTime is unsupported in this build")
}

165
integration/remote/util/util_windows.go Normal file
View File

@ -0,0 +1,165 @@
// +build windows
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
/*
Copyright 2017 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package util
import (
"context"
"fmt"
"net"
"net/url"
"strings"
"syscall"
"time"
"github.com/Microsoft/go-winio"
)
const (
tcpProtocol = "tcp"
npipeProtocol = "npipe"
)
// CreateListener creates a listener on the specified endpoint.
func CreateListener(endpoint string) (net.Listener, error) {
protocol, addr, err := parseEndpoint(endpoint)
if err != nil {
return nil, err
}
switch protocol {
case tcpProtocol:
return net.Listen(tcpProtocol, addr)
case npipeProtocol:
return winio.ListenPipe(addr, nil)
default:
return nil, fmt.Errorf("only support tcp and npipe endpoint")
}
}
// GetAddressAndDialer returns the address parsed from the given endpoint and a context dialer.
func GetAddressAndDialer(endpoint string) (string, func(ctx context.Context, addr string) (net.Conn, error), error) {
protocol, addr, err := parseEndpoint(endpoint)
if err != nil {
return "", nil, err
}
if protocol == tcpProtocol {
return addr, tcpDial, nil
}
if protocol == npipeProtocol {
return addr, npipeDial, nil
}
return "", nil, fmt.Errorf("only support tcp and npipe endpoint")
}
func tcpDial(ctx context.Context, addr string) (net.Conn, error) {
return (&net.Dialer{}).DialContext(ctx, tcpProtocol, addr)
}
func npipeDial(ctx context.Context, addr string) (net.Conn, error) {
return winio.DialPipeContext(ctx, addr)
}
func parseEndpoint(endpoint string) (string, string, error) {
// url.Parse doesn't recognize \, so replace with / first.
endpoint = strings.Replace(endpoint, "\\", "/", -1)
u, err := url.Parse(endpoint)
if err != nil {
return "", "", err
}
if u.Scheme == "tcp" {
return "tcp", u.Host, nil
} else if u.Scheme == "npipe" {
if strings.HasPrefix(u.Path, "//./pipe") {
return "npipe", u.Path, nil
}
// fallback host if not provided.
host := u.Host
if host == "" {
host = "."
}
return "npipe", fmt.Sprintf("//%s%s", host, u.Path), nil
} else if u.Scheme == "" {
return "", "", fmt.Errorf("Using %q as endpoint is deprecated, please consider using full url format", endpoint)
} else {
return u.Scheme, "", fmt.Errorf("protocol %q not supported", u.Scheme)
}
}
var tickCount = syscall.NewLazyDLL("kernel32.dll").NewProc("GetTickCount64")
// GetBootTime returns the time at which the machine was started, truncated to the nearest second
func GetBootTime() (time.Time, error) {
currentTime := time.Now()
output, _, err := tickCount.Call()
if errno, ok := err.(syscall.Errno); !ok || errno != 0 {
return time.Time{}, err
}
return currentTime.Add(-time.Duration(output) * time.Millisecond).Truncate(time.Second), nil
}
// IsUnixDomainSocket returns whether a given file is a AF_UNIX socket file
func IsUnixDomainSocket(filePath string) (bool, error) {
// Due to the absence of golang support for os.ModeSocket in Windows (https://github.com/golang/go/issues/33357)
// we need to dial the file and check if we receive an error to determine if a file is Unix Domain Socket file.
// Note that querrying for the Reparse Points (https://docs.microsoft.com/en-us/windows/win32/fileio/reparse-points)
// for the file (using FSCTL_GET_REPARSE_POINT) and checking for reparse tag: reparseTagSocket
// does NOT work in 1809 if the socket file is created within a bind mounted directory by a container
// and the FSCTL is issued in the host by the kubelet.
c, err := net.Dial("unix", filePath)
if err == nil {
c.Close()
return true, nil
}
return false, nil
}
// NormalizePath converts FS paths returned by certain go frameworks (like fsnotify)
// to native Windows paths that can be passed to Windows specific code
func NormalizePath(path string) string {
path = strings.ReplaceAll(path, "/", "\\")
if strings.HasPrefix(path, "\\") {
path = "c:" + path
}
return path
}

107
integration/remote/utils.go Normal file
View File

@ -0,0 +1,107 @@
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
/*
Copyright 2016 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package remote
import (
"context"
"fmt"
"time"
runtimeapi "k8s.io/cri-api/pkg/apis/runtime/v1alpha2"
)
// maxMsgSize use 16MB as the default message size limit.
// grpc library default is 4MB
const maxMsgSize = 1024 * 1024 * 16
// getContextWithTimeout returns a context with timeout.
func getContextWithTimeout(timeout time.Duration) (context.Context, context.CancelFunc) {
return context.WithTimeout(context.Background(), timeout)
}
// getContextWithCancel returns a context with cancel.
func getContextWithCancel() (context.Context, context.CancelFunc) {
return context.WithCancel(context.Background())
}
// verifySandboxStatus verified whether all required fields are set in PodSandboxStatus.
func verifySandboxStatus(status *runtimeapi.PodSandboxStatus) error {
if status.Id == "" {
return fmt.Errorf("Id is not set")
}
if status.Metadata == nil {
return fmt.Errorf("Metadata is not set")
}
metadata := status.Metadata
if metadata.Name == "" || metadata.Namespace == "" || metadata.Uid == "" {
return fmt.Errorf("Name, Namespace or Uid is not in metadata %q", metadata)
}
if status.CreatedAt == 0 {
return fmt.Errorf("CreatedAt is not set")
}
return nil
}
// verifyContainerStatus verified whether all required fields are set in ContainerStatus.
func verifyContainerStatus(status *runtimeapi.ContainerStatus) error {
if status.Id == "" {
return fmt.Errorf("Id is not set")
}
if status.Metadata == nil {
return fmt.Errorf("Metadata is not set")
}
metadata := status.Metadata
if metadata.Name == "" {
return fmt.Errorf("Name is not in metadata %q", metadata)
}
if status.CreatedAt == 0 {
return fmt.Errorf("CreatedAt is not set")
}
if status.Image == nil || status.Image.Image == "" {
return fmt.Errorf("Image is not set")
}
if status.ImageRef == "" {
return fmt.Errorf("ImageRef is not set")
}
return nil
}

201
integration/restart_test.go Normal file
View File

@ -0,0 +1,201 @@
// +build linux
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package integration
import (
"sort"
"testing"
"github.com/containerd/containerd"
"github.com/containerd/containerd/errdefs"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"golang.org/x/net/context"
runtime "k8s.io/cri-api/pkg/apis/runtime/v1alpha2"
)
// Restart test must run sequentially.
func TestContainerdRestart(t *testing.T) {
type container struct {
name string
id string
state runtime.ContainerState
}
type sandbox struct {
name string
id string
state runtime.PodSandboxState
containers []container
}
ctx := context.Background()
sandboxNS := "restart-containerd"
sandboxes := []sandbox{
{
name: "ready-sandbox",
state: runtime.PodSandboxState_SANDBOX_READY,
containers: []container{
{
name: "created-container",
state: runtime.ContainerState_CONTAINER_CREATED,
},
{
name: "running-container",
state: runtime.ContainerState_CONTAINER_RUNNING,
},
{
name: "exited-container",
state: runtime.ContainerState_CONTAINER_EXITED,
},
},
},
{
name: "notready-sandbox",
state: runtime.PodSandboxState_SANDBOX_NOTREADY,
containers: []container{
{
name: "created-container",
state: runtime.ContainerState_CONTAINER_CREATED,
},
{
name: "running-container",
state: runtime.ContainerState_CONTAINER_RUNNING,
},
{
name: "exited-container",
state: runtime.ContainerState_CONTAINER_EXITED,
},
},
},
}
t.Logf("Make sure no sandbox is running before test")
existingSandboxes, err := runtimeService.ListPodSandbox(&runtime.PodSandboxFilter{})
require.NoError(t, err)
require.Empty(t, existingSandboxes)
t.Logf("Start test sandboxes and containers")
for i := range sandboxes {
s := &sandboxes[i]
sbCfg := PodSandboxConfig(s.name, sandboxNS)
sid, err := runtimeService.RunPodSandbox(sbCfg, *runtimeHandler)
require.NoError(t, err)
defer func() {
// Make sure the sandbox is cleaned up in any case.
runtimeService.StopPodSandbox(sid)
runtimeService.RemovePodSandbox(sid)
}()
s.id = sid
for j := range s.containers {
c := &s.containers[j]
cfg := ContainerConfig(c.name, pauseImage,
// Set pid namespace as per container, so that container won't die
// when sandbox container is killed.
WithPidNamespace(runtime.NamespaceMode_CONTAINER),
)
cid, err := runtimeService.CreateContainer(sid, cfg, sbCfg)
require.NoError(t, err)
// Reply on sandbox cleanup.
c.id = cid
switch c.state {
case runtime.ContainerState_CONTAINER_CREATED:
case runtime.ContainerState_CONTAINER_RUNNING:
require.NoError(t, runtimeService.StartContainer(cid))
case runtime.ContainerState_CONTAINER_EXITED:
require.NoError(t, runtimeService.StartContainer(cid))
require.NoError(t, runtimeService.StopContainer(cid, 10))
}
}
if s.state == runtime.PodSandboxState_SANDBOX_NOTREADY {
cntr, err := containerdClient.LoadContainer(ctx, sid)
require.NoError(t, err)
task, err := cntr.Task(ctx, nil)
require.NoError(t, err)
_, err = task.Delete(ctx, containerd.WithProcessKill)
if err != nil {
require.True(t, errdefs.IsNotFound(err))
}
}
}
t.Logf("Pull test images")
for _, image := range []string{"busybox", "alpine"} {
img, err := imageService.PullImage(&runtime.ImageSpec{Image: image}, nil, nil)
require.NoError(t, err)
defer func() {
assert.NoError(t, imageService.RemoveImage(&runtime.ImageSpec{Image: img}))
}()
}
imagesBeforeRestart, err := imageService.ListImages(nil)
assert.NoError(t, err)
t.Logf("Restart containerd")
RestartContainerd(t)
t.Logf("Check sandbox and container state after restart")
loadedSandboxes, err := runtimeService.ListPodSandbox(&runtime.PodSandboxFilter{})
require.NoError(t, err)
assert.Len(t, loadedSandboxes, len(sandboxes))
loadedContainers, err := runtimeService.ListContainers(&runtime.ContainerFilter{})
require.NoError(t, err)
assert.Len(t, loadedContainers, len(sandboxes)*3)
for _, s := range sandboxes {
for _, loaded := range loadedSandboxes {
if s.id == loaded.Id {
assert.Equal(t, s.state, loaded.State)
break
}
}
for _, c := range s.containers {
for _, loaded := range loadedContainers {
if c.id == loaded.Id {
assert.Equal(t, c.state, loaded.State)
break
}
}
}
}
t.Logf("Should be able to stop and remove sandbox after restart")
for _, s := range sandboxes {
assert.NoError(t, runtimeService.StopPodSandbox(s.id))
assert.NoError(t, runtimeService.RemovePodSandbox(s.id))
}
t.Logf("Should recover all images")
imagesAfterRestart, err := imageService.ListImages(nil)
assert.NoError(t, err)
assert.Equal(t, len(imagesBeforeRestart), len(imagesAfterRestart))
for _, i1 := range imagesBeforeRestart {
found := false
for _, i2 := range imagesAfterRestart {
if i1.Id == i2.Id {
sort.Strings(i1.RepoTags)
sort.Strings(i1.RepoDigests)
sort.Strings(i2.RepoTags)
sort.Strings(i2.RepoDigests)
assert.Equal(t, i1, i2)
found = true
break
}
}
assert.True(t, found, "should find image %+v", i1)
}
}
// TODO: Add back the unknown state test.

52
integration/runtime_handler_test.go Normal file
View File

@ -0,0 +1,52 @@
// +build linux
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package integration
import (
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
runtime "k8s.io/cri-api/pkg/apis/runtime/v1alpha2"
)
func TestRuntimeHandler(t *testing.T) {
t.Logf("Create a sandbox")
sbConfig := PodSandboxConfig("sandbox", "test-runtime-handler")
t.Logf("the --runtime-handler flag value is: %s", *runtimeHandler)
sb, err := runtimeService.RunPodSandbox(sbConfig, *runtimeHandler)
require.NoError(t, err)
defer func() {
// Make sure the sandbox is cleaned up in any case.
runtimeService.StopPodSandbox(sb)
runtimeService.RemovePodSandbox(sb)
}()
t.Logf("Verify runtimeService.PodSandboxStatus sets RuntimeHandler")
sbStatus, err := runtimeService.PodSandboxStatus(sb)
require.NoError(t, err)
t.Logf("runtimeService.PodSandboxStatus sets RuntimeHandler to %s", sbStatus.RuntimeHandler)
assert.Equal(t, *runtimeHandler, sbStatus.RuntimeHandler)
t.Logf("Verify runtimeService.ListPodSandbox sets RuntimeHandler")
sandboxes, err := runtimeService.ListPodSandbox(&runtime.PodSandboxFilter{})
require.NoError(t, err)
t.Logf("runtimeService.ListPodSandbox sets RuntimeHandler to %s", sbStatus.RuntimeHandler)
assert.Equal(t, *runtimeHandler, sandboxes[0].RuntimeHandler)
}

126
integration/sandbox_clean_remove_test.go Normal file
View File

@ -0,0 +1,126 @@
// +build linux
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package integration
import (
"io/ioutil"
"os"
"path/filepath"
"strings"
"testing"
"time"
runtimespec "github.com/opencontainers/runtime-spec/specs-go"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"golang.org/x/sys/unix"
runtime "k8s.io/cri-api/pkg/apis/runtime/v1alpha2"
)
func TestSandboxRemoveWithoutIPLeakage(t *testing.T) {
const hostLocalCheckpointDir = "/var/lib/cni"
t.Logf("Make sure host-local ipam is in use")
config, err := CRIConfig()
require.NoError(t, err)
fs, err := ioutil.ReadDir(config.NetworkPluginConfDir)
require.NoError(t, err)
require.NotEmpty(t, fs)
f := filepath.Join(config.NetworkPluginConfDir, fs[0].Name())
cniConfig, err := ioutil.ReadFile(f)
require.NoError(t, err)
if !strings.Contains(string(cniConfig), "host-local") {
t.Skip("host-local ipam is not in use")
}
t.Logf("Create a sandbox")
sbConfig := PodSandboxConfig("sandbox", "remove-without-ip-leakage")
sb, err := runtimeService.RunPodSandbox(sbConfig, *runtimeHandler)
require.NoError(t, err)
defer func() {
// Make sure the sandbox is cleaned up in any case.
runtimeService.StopPodSandbox(sb)
runtimeService.RemovePodSandbox(sb)
}()
t.Logf("Get pod information")
status, info, err := SandboxInfo(sb)
require.NoError(t, err)
ip := status.GetNetwork().GetIp()
require.NotEmpty(t, ip)
require.NotNil(t, info.RuntimeSpec.Linux)
var netNS string
for _, n := range info.RuntimeSpec.Linux.Namespaces {
if n.Type == runtimespec.NetworkNamespace {
netNS = n.Path
}
}
require.NotEmpty(t, netNS, "network namespace should be set")
t.Logf("Should be able to find the pod ip in host-local checkpoint")
checkIP := func(ip string) bool {
found := false
filepath.Walk(hostLocalCheckpointDir, func(_ string, info os.FileInfo, _ error) error {
if info != nil && info.Name() == ip {
found = true
}
return nil
})
return found
}
require.True(t, checkIP(ip))
t.Logf("Kill sandbox container")
require.NoError(t, KillPid(int(info.Pid)))
t.Logf("Unmount network namespace")
require.NoError(t, unix.Unmount(netNS, unix.MNT_DETACH))
t.Logf("Network namespace should be closed")
_, info, err = SandboxInfo(sb)
require.NoError(t, err)
assert.True(t, info.NetNSClosed)
t.Logf("Remove network namespace")
require.NoError(t, os.RemoveAll(netNS))
t.Logf("Network namespace should still be closed")
_, info, err = SandboxInfo(sb)
require.NoError(t, err)
assert.True(t, info.NetNSClosed)
t.Logf("Sandbox state should be NOTREADY")
assert.NoError(t, Eventually(func() (bool, error) {
status, err := runtimeService.PodSandboxStatus(sb)
if err != nil {
return false, err
}
return status.GetState() == runtime.PodSandboxState_SANDBOX_NOTREADY, nil
}, time.Second, 30*time.Second), "sandbox state should become NOTREADY")
t.Logf("Should still be able to find the pod ip in host-local checkpoint")
assert.True(t, checkIP(ip))
t.Logf("Should be able to stop and remove the sandbox")
assert.NoError(t, runtimeService.StopPodSandbox(sb))
assert.NoError(t, runtimeService.RemovePodSandbox(sb))
t.Logf("Should not be able to find the pod ip in host-local checkpoint")
assert.False(t, checkIP(ip))
}

160
integration/truncindex_test.go Normal file
View File

@ -0,0 +1,160 @@
// +build linux
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package integration
import (
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
runtimeapi "k8s.io/cri-api/pkg/apis/runtime/v1alpha2"
)
func genTruncIndex(normalName string) string {
return normalName[:(len(normalName)+1)/2]
}
func TestTruncIndex(t *testing.T) {
sbConfig := PodSandboxConfig("sandbox", "truncindex")
t.Logf("Pull an image")
const appImage = "busybox"
imgID, err := imageService.PullImage(&runtimeapi.ImageSpec{Image: appImage}, nil, sbConfig)
require.NoError(t, err)
imgTruncID := genTruncIndex(imgID)
defer func() {
assert.NoError(t, imageService.RemoveImage(&runtimeapi.ImageSpec{Image: imgTruncID}))
}()
t.Logf("Get image status by truncindex, truncID: %s", imgTruncID)
res, err := imageService.ImageStatus(&runtimeapi.ImageSpec{Image: imgTruncID})
require.NoError(t, err)
require.NotEqual(t, nil, res)
assert.Equal(t, imgID, res.Id)
// TODO(yanxuean): for failure test case where there are two images with the same truncindex.
// if you add n images at least two will share the same leading digit.
// "sha256:n" where n is the a number from 0-9 where two images have the same trunc,
// for example sha256:9
// https://github.com/containerd/cri/pull/352
// I am thinking how I get the two image which have same trunc.
// TODO(yanxuean): add test case for ListImages
t.Logf("Create a sandbox")
sb, err := runtimeService.RunPodSandbox(sbConfig, *runtimeHandler)
require.NoError(t, err)
sbTruncIndex := genTruncIndex(sb)
var hasStoppedSandbox bool
defer func() {
// The 2th StopPodSandbox will fail, the 2th RemovePodSandbox will success.
if !hasStoppedSandbox {
assert.NoError(t, runtimeService.StopPodSandbox(sbTruncIndex))
}
assert.NoError(t, runtimeService.RemovePodSandbox(sbTruncIndex))
}()
t.Logf("Get sandbox status by truncindex")
sbStatus, err := runtimeService.PodSandboxStatus(sbTruncIndex)
require.NoError(t, err)
assert.Equal(t, sb, sbStatus.Id)
t.Logf("Forward port for sandbox by truncindex")
_, err = runtimeService.PortForward(&runtimeapi.PortForwardRequest{PodSandboxId: sbTruncIndex, Port: []int32{80}})
assert.NoError(t, err)
// TODO(yanxuean): add test case for ListPodSandbox
t.Logf("Create a container")
cnConfig := ContainerConfig(
"containerTruncIndex",
appImage,
WithCommand("top"),
)
cn, err := runtimeService.CreateContainer(sbTruncIndex, cnConfig, sbConfig)
require.NoError(t, err)
cnTruncIndex := genTruncIndex(cn)
defer func() {
// the 2th RemovePodSandbox will success.
assert.NoError(t, runtimeService.RemoveContainer(cnTruncIndex))
}()
t.Logf("Get container status by truncindex")
cStatus, err := runtimeService.ContainerStatus(cnTruncIndex)
require.NoError(t, err)
assert.Equal(t, cn, cStatus.Id)
t.Logf("Start the container")
require.NoError(t, runtimeService.StartContainer(cnTruncIndex))
var hasStoppedContainer bool
defer func() {
// The 2th StopPodSandbox will fail
if !hasStoppedContainer {
assert.NoError(t, runtimeService.StopContainer(cnTruncIndex, 10))
}
}()
t.Logf("Stats the container")
cStats, err := runtimeService.ContainerStats(cnTruncIndex)
require.NoError(t, err)
assert.Equal(t, cn, cStats.Attributes.Id)
t.Logf("Update container memory limit after started")
err = runtimeService.UpdateContainerResources(cnTruncIndex, &runtimeapi.LinuxContainerResources{
MemoryLimitInBytes: 50 * 1024 * 1024,
})
assert.NoError(t, err)
t.Logf("Execute cmd in container")
execReq := &runtimeapi.ExecRequest{
ContainerId: cnTruncIndex,
Cmd: []string{"pwd"},
Stdout: true,
}
_, err = runtimeService.Exec(execReq)
assert.NoError(t, err)
t.Logf("Execute cmd in container by sync")
_, _, err = runtimeService.ExecSync(cnTruncIndex, []string{"pwd"}, 10)
assert.NoError(t, err)
// TODO(yanxuean): add test case for ListContainers
t.Logf("Get a non exist container status by truncindex")
err = runtimeService.StopContainer(cnTruncIndex, 10)
assert.NoError(t, err)
if err == nil {
hasStoppedContainer = true
}
_, err = runtimeService.ContainerStats(cnTruncIndex)
assert.Error(t, err)
assert.NoError(t, runtimeService.RemoveContainer(cnTruncIndex))
_, err = runtimeService.ContainerStatus(cnTruncIndex)
assert.Error(t, err)
t.Logf("Get a non exist sandbox status by truncindex")
err = runtimeService.StopPodSandbox(sbTruncIndex)
assert.NoError(t, err)
if err == nil {
hasStoppedSandbox = true
}
assert.NoError(t, runtimeService.RemovePodSandbox(sbTruncIndex))
_, err = runtimeService.PodSandboxStatus(sbTruncIndex)
assert.Error(t, err)
}

60
integration/util/boottime_util_darwin.go Normal file
View File

@ -0,0 +1,60 @@
// +build darwin
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
/*
Copyright 2018 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package util
import (
"fmt"
"syscall"
"time"
"unsafe"
"golang.org/x/sys/unix"
)
// GetBootTime returns the time at which the machine was started, truncated to the nearest second
func GetBootTime() (time.Time, error) {
output, err := unix.SysctlRaw("kern.boottime")
if err != nil {
return time.Time{}, err
}
var timeval syscall.Timeval
if len(output) != int(unsafe.Sizeof(timeval)) {
return time.Time{}, fmt.Errorf("unexpected output when calling syscall kern.bootime. Expected len(output) to be %v, but got %v",
int(unsafe.Sizeof(timeval)), len(output))
}
timeval = *(*syscall.Timeval)(unsafe.Pointer(&output[0]))
sec, nsec := timeval.Unix()
return time.Unix(sec, nsec).Truncate(time.Second), nil
}

52
integration/util/boottime_util_linux.go Normal file
View File

@ -0,0 +1,52 @@
// +build freebsd linux
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
/*
Copyright 2018 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package util
import (
"fmt"
"time"
"golang.org/x/sys/unix"
)
// GetBootTime returns the time at which the machine was started, truncated to the nearest second
func GetBootTime() (time.Time, error) {
currentTime := time.Now()
var info unix.Sysinfo_t
if err := unix.Sysinfo(&info); err != nil {
return time.Time{}, fmt.Errorf("error getting system uptime: %s", err)
}
return currentTime.Add(-time.Duration(info.Uptime) * time.Second).Truncate(time.Second), nil
}

34
integration/util/doc.go Normal file
View File

@ -0,0 +1,34 @@
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
/*
Copyright 2015 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// Package util holds utility functions.
package util

43
integration/util/util.go Normal file
View File

@ -0,0 +1,43 @@
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
/*
Copyright 2017 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package util
import (
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
// FromApiserverCache modifies <opts> so that the GET request will
// be served from apiserver cache instead of from etcd.
func FromApiserverCache(opts *metav1.GetOptions) {
opts.ResourceVersion = "0"
}

170
integration/util/util_unix.go Normal file
View File

@ -0,0 +1,170 @@
// +build freebsd linux darwin
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
/*
Copyright 2017 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package util
import (
"context"
"fmt"
"io/ioutil"
"net"
"net/url"
"os"
"path/filepath"
"golang.org/x/sys/unix"
"k8s.io/klog/v2"
)
const (
// unixProtocol is the network protocol of unix socket.
unixProtocol = "unix"
)
// CreateListener creates a listener on the specified endpoint.
func CreateListener(endpoint string) (net.Listener, error) {
protocol, addr, err := parseEndpointWithFallbackProtocol(endpoint, unixProtocol)
if err != nil {
return nil, err
}
if protocol != unixProtocol {
return nil, fmt.Errorf("only support unix socket endpoint")
}
// Unlink to cleanup the previous socket file.
err = unix.Unlink(addr)
if err != nil && !os.IsNotExist(err) {
return nil, fmt.Errorf("failed to unlink socket file %q: %v", addr, err)
}
if err := os.MkdirAll(filepath.Dir(addr), 0750); err != nil {
return nil, fmt.Errorf("error creating socket directory %q: %v", filepath.Dir(addr), err)
}
// Create the socket on a tempfile and move it to the destination socket to handle improprer cleanup
file, err := ioutil.TempFile(filepath.Dir(addr), "")
if err != nil {
return nil, fmt.Errorf("failed to create temporary file: %v", err)
}
if err := os.Remove(file.Name()); err != nil {
return nil, fmt.Errorf("failed to remove temporary file: %v", err)
}
l, err := net.Listen(protocol, file.Name())
if err != nil {
return nil, err
}
if err = os.Rename(file.Name(), addr); err != nil {
return nil, fmt.Errorf("failed to move temporary file to addr %q: %v", addr, err)
}
return l, nil
}
// GetAddressAndDialer returns the address parsed from the given endpoint and a context dialer.
func GetAddressAndDialer(endpoint string) (string, func(ctx context.Context, addr string) (net.Conn, error), error) {
protocol, addr, err := parseEndpointWithFallbackProtocol(endpoint, unixProtocol)
if err != nil {
return "", nil, err
}
if protocol != unixProtocol {
return "", nil, fmt.Errorf("only support unix socket endpoint")
}
return addr, dial, nil
}
func dial(ctx context.Context, addr string) (net.Conn, error) {
return (&net.Dialer{}).DialContext(ctx, unixProtocol, addr)
}
func parseEndpointWithFallbackProtocol(endpoint string, fallbackProtocol string) (protocol string, addr string, err error) {
if protocol, addr, err = parseEndpoint(endpoint); err != nil && protocol == "" {
fallbackEndpoint := fallbackProtocol + "://" + endpoint
protocol, addr, err = parseEndpoint(fallbackEndpoint)
if err == nil {
klog.Warningf("Using %q as endpoint is deprecated, please consider using full url format %q.", endpoint, fallbackEndpoint)
}
}
return
}
func parseEndpoint(endpoint string) (string, string, error) {
u, err := url.Parse(endpoint)
if err != nil {
return "", "", err
}
switch u.Scheme {
case "tcp":
return "tcp", u.Host, nil
case "unix":
return "unix", u.Path, nil
case "":
return "", "", fmt.Errorf("using %q as endpoint is deprecated, please consider using full url format", endpoint)
default:
return u.Scheme, "", fmt.Errorf("protocol %q not supported", u.Scheme)
}
}
// LocalEndpoint returns the full path to a unix socket at the given endpoint
func LocalEndpoint(path, file string) (string, error) {
u := url.URL{
Scheme: unixProtocol,
Path: path,
}
return filepath.Join(u.String(), file+".sock"), nil
}
// IsUnixDomainSocket returns whether a given file is a AF_UNIX socket file
func IsUnixDomainSocket(filePath string) (bool, error) {
fi, err := os.Stat(filePath)
if err != nil {
return false, fmt.Errorf("stat file %s failed: %v", filePath, err)
}
if fi.Mode()&os.ModeSocket == 0 {
return false, nil
}
return true, nil
}
// NormalizePath is a no-op for Linux for now
func NormalizePath(path string) string {
return path
}

71
integration/util/util_unsupported.go Normal file
View File

@ -0,0 +1,71 @@
// +build !freebsd,!linux,!windows,!darwin
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
/*
Copyright 2017 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package util
import (
"context"
"fmt"
"net"
"time"
)
// CreateListener creates a listener on the specified endpoint.
func CreateListener(endpoint string) (net.Listener, error) {
return nil, fmt.Errorf("CreateListener is unsupported in this build")
}
// GetAddressAndDialer returns the address parsed from the given endpoint and a context dialer.
func GetAddressAndDialer(endpoint string) (string, func(ctx context.Context, addr string) (net.Conn, error), error) {
return "", nil, fmt.Errorf("GetAddressAndDialer is unsupported in this build")
}
// LockAndCheckSubPath empty implementation
func LockAndCheckSubPath(volumePath, subPath string) ([]uintptr, error) {
return []uintptr{}, nil
}
// UnlockPath empty implementation
func UnlockPath(fileHandles []uintptr) {
}
// LocalEndpoint empty implementation
func LocalEndpoint(path, file string) (string, error) {
return "", fmt.Errorf("LocalEndpoints are unsupported in this build")
}
// GetBootTime empty implementation
func GetBootTime() (time.Time, error) {
return time.Time{}, fmt.Errorf("GetBootTime is unsupported in this build")
}

170
integration/util/util_windows.go Normal file
View File

@ -0,0 +1,170 @@
// +build windows
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
/*
Copyright 2017 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package util
import (
"context"
"fmt"
"net"
"net/url"
"strings"
"syscall"
"time"
"github.com/Microsoft/go-winio"
)
const (
tcpProtocol = "tcp"
npipeProtocol = "npipe"
)
// CreateListener creates a listener on the specified endpoint.
func CreateListener(endpoint string) (net.Listener, error) {
protocol, addr, err := parseEndpoint(endpoint)
if err != nil {
return nil, err
}
switch protocol {
case tcpProtocol:
return net.Listen(tcpProtocol, addr)
case npipeProtocol:
return winio.ListenPipe(addr, nil)
default:
return nil, fmt.Errorf("only support tcp and npipe endpoint")
}
}
// GetAddressAndDialer returns the address parsed from the given endpoint and a context dialer.
func GetAddressAndDialer(endpoint string) (string, func(ctx context.Context, addr string) (net.Conn, error), error) {
protocol, addr, err := parseEndpoint(endpoint)
if err != nil {
return "", nil, err
}
if protocol == tcpProtocol {
return addr, tcpDial, nil
}
if protocol == npipeProtocol {
return addr, npipeDial, nil
}
return "", nil, fmt.Errorf("only support tcp and npipe endpoint")
}
func tcpDial(ctx context.Context, addr string) (net.Conn, error) {
return (&net.Dialer{}).DialContext(ctx, tcpProtocol, addr)
}
func npipeDial(ctx context.Context, addr string) (net.Conn, error) {
return winio.DialPipeContext(ctx, addr)
}
func parseEndpoint(endpoint string) (string, string, error) {
// url.Parse doesn't recognize \, so replace with / first.
endpoint = strings.Replace(endpoint, "\\", "/", -1)
u, err := url.Parse(endpoint)
if err != nil {
return "", "", err
}
if u.Scheme == "tcp" {
return "tcp", u.Host, nil
} else if u.Scheme == "npipe" {
if strings.HasPrefix(u.Path, "//./pipe") {
return "npipe", u.Path, nil
}
// fallback host if not provided.
host := u.Host
if host == "" {
host = "."
}
return "npipe", fmt.Sprintf("//%s%s", host, u.Path), nil
} else if u.Scheme == "" {
return "", "", fmt.Errorf("Using %q as endpoint is deprecated, please consider using full url format", endpoint)
} else {
return u.Scheme, "", fmt.Errorf("protocol %q not supported", u.Scheme)
}
}
// LocalEndpoint empty implementation
func LocalEndpoint(path, file string) (string, error) {
return "", fmt.Errorf("LocalEndpoints are unsupported in this build")
}
var tickCount = syscall.NewLazyDLL("kernel32.dll").NewProc("GetTickCount64")
// GetBootTime returns the time at which the machine was started, truncated to the nearest second
func GetBootTime() (time.Time, error) {
currentTime := time.Now()
output, _, err := tickCount.Call()
if errno, ok := err.(syscall.Errno); !ok || errno != 0 {
return time.Time{}, err
}
return currentTime.Add(-time.Duration(output) * time.Millisecond).Truncate(time.Second), nil
}
// IsUnixDomainSocket returns whether a given file is a AF_UNIX socket file
func IsUnixDomainSocket(filePath string) (bool, error) {
// Due to the absence of golang support for os.ModeSocket in Windows (https://github.com/golang/go/issues/33357)
// we need to dial the file and check if we receive an error to determine if a file is Unix Domain Socket file.
// Note that querrying for the Reparse Points (https://docs.microsoft.com/en-us/windows/win32/fileio/reparse-points)
// for the file (using FSCTL_GET_REPARSE_POINT) and checking for reparse tag: reparseTagSocket
// does NOT work in 1809 if the socket file is created within a bind mounted directory by a container
// and the FSCTL is issued in the host by the kubelet.
c, err := net.Dial("unix", filePath)
if err == nil {
c.Close()
return true, nil
}
return false, nil
}
// NormalizePath converts FS paths returned by certain go frameworks (like fsnotify)
// to native Windows paths that can be passed to Windows specific code
func NormalizePath(path string) string {
path = strings.ReplaceAll(path, "/", "\\")
if strings.HasPrefix(path, "\\") {
path = "c:" + path
}
return path
}

140
integration/volume_copy_up_test.go Normal file
View File

@ -0,0 +1,140 @@
// +build linux
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package integration
import (
"fmt"
"os/exec"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
runtime "k8s.io/cri-api/pkg/apis/runtime/v1alpha2"
)
func TestVolumeCopyUp(t *testing.T) {
const (
testImage = "gcr.io/k8s-cri-containerd/volume-copy-up:1.0"
execTimeout = time.Minute
)
t.Logf("Create a sandbox")
sbConfig := PodSandboxConfig("sandbox", "volume-copy-up")
sb, err := runtimeService.RunPodSandbox(sbConfig, *runtimeHandler)
require.NoError(t, err)
defer func() {
assert.NoError(t, runtimeService.StopPodSandbox(sb))
assert.NoError(t, runtimeService.RemovePodSandbox(sb))
}()
t.Logf("Pull test image")
_, err = imageService.PullImage(&runtime.ImageSpec{Image: testImage}, nil, sbConfig)
require.NoError(t, err)
t.Logf("Create a container with volume-copy-up test image")
cnConfig := ContainerConfig(
"container",
testImage,
WithCommand("tail", "-f", "/dev/null"),
)
cn, err := runtimeService.CreateContainer(sb, cnConfig, sbConfig)
require.NoError(t, err)
t.Logf("Start the container")
require.NoError(t, runtimeService.StartContainer(cn))
// gcr.io/k8s-cri-containerd/volume-copy-up:1.0 contains a test_dir
// volume, which contains a test_file with content "test_content".
t.Logf("Check whether volume contains the test file")
stdout, stderr, err := runtimeService.ExecSync(cn, []string{
"cat",
"/test_dir/test_file",
}, execTimeout)
require.NoError(t, err)
assert.Empty(t, stderr)
assert.Equal(t, "test_content\n", string(stdout))
t.Logf("Check host path of the volume")
hostCmd := fmt.Sprintf("find %s/containers/%s/volumes/*/test_file | xargs cat", *criRoot, cn)
output, err := exec.Command("sh", "-c", hostCmd).CombinedOutput()
require.NoError(t, err)
assert.Equal(t, "test_content\n", string(output))
t.Logf("Update volume from inside the container")
_, _, err = runtimeService.ExecSync(cn, []string{
"sh",
"-c",
"echo new_content > /test_dir/test_file",
}, execTimeout)
require.NoError(t, err)
t.Logf("Check whether host path of the volume is updated")
output, err = exec.Command("sh", "-c", hostCmd).CombinedOutput()
require.NoError(t, err)
assert.Equal(t, "new_content\n", string(output))
}
func TestVolumeOwnership(t *testing.T) {
const (
testImage = "gcr.io/k8s-cri-containerd/volume-ownership:1.0"
execTimeout = time.Minute
)
t.Logf("Create a sandbox")
sbConfig := PodSandboxConfig("sandbox", "volume-ownership")
sb, err := runtimeService.RunPodSandbox(sbConfig, *runtimeHandler)
require.NoError(t, err)
defer func() {
assert.NoError(t, runtimeService.StopPodSandbox(sb))
assert.NoError(t, runtimeService.RemovePodSandbox(sb))
}()
t.Logf("Pull test image")
_, err = imageService.PullImage(&runtime.ImageSpec{Image: testImage}, nil, sbConfig)
require.NoError(t, err)
t.Logf("Create a container with volume-ownership test image")
cnConfig := ContainerConfig(
"container",
testImage,
WithCommand("tail", "-f", "/dev/null"),
)
cn, err := runtimeService.CreateContainer(sb, cnConfig, sbConfig)
require.NoError(t, err)
t.Logf("Start the container")
require.NoError(t, runtimeService.StartContainer(cn))
// gcr.io/k8s-cri-containerd/volume-ownership:1.0 contains a test_dir
// volume, which is owned by nobody:nogroup.
t.Logf("Check ownership of test directory inside container")
stdout, stderr, err := runtimeService.ExecSync(cn, []string{
"stat", "-c", "%U:%G", "/test_dir",
}, execTimeout)
require.NoError(t, err)
assert.Empty(t, stderr)
assert.Equal(t, "nobody:nogroup\n", string(stdout))
t.Logf("Check ownership of test directory on the host")
hostCmd := fmt.Sprintf("find %s/containers/%s/volumes/* | xargs stat -c %%U:%%G", *criRoot, cn)
output, err := exec.Command("sh", "-c", hostCmd).CombinedOutput()
require.NoError(t, err)
assert.Equal(t, "nobody:nogroup\n", string(output))
}

50
pkg/annotations/annotations.go Normal file
View File

@ -0,0 +1,50 @@
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package annotations
// ContainerType values
// Following OCI annotations are used by katacontainers now.
// We'll switch to standard secure pod API after it is defined in CRI.
const (
// ContainerTypeSandbox represents a pod sandbox container
ContainerTypeSandbox = "sandbox"
// ContainerTypeContainer represents a container running within a pod
ContainerTypeContainer = "container"
// ContainerType is the container type (sandbox or container) annotation
ContainerType = "io.kubernetes.cri.container-type"
// SandboxID is the sandbox ID annotation
SandboxID = "io.kubernetes.cri.sandbox-id"
// SandboxLogDir is the pod log directory annotation.
// If the sandbox needs to generate any log, it will put it into this directory.
// Kubelet will be responsible for:
// 1) Monitoring the disk usage of the log, and including it as part of the pod
// ephemeral storage usage.
// 2) Cleaning up the logs when the pod is deleted.
// NOTE: Kubelet is not responsible for rotating the logs.
SandboxLogDir = "io.kubernetes.cri.sandbox-log-directory"
// UntrustedWorkload is the sandbox annotation for untrusted workload. Untrusted
// workload can only run on dedicated runtime for untrusted workload.
UntrustedWorkload = "io.kubernetes.cri.untrusted-workload"
// containerName is the name of the container in the pod
ContainerName = "io.kubernetes.cri.container-name"
)

394
pkg/api/runtimeoptions/v1/api.pb.go Normal file
View File

@ -0,0 +1,394 @@
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// Code generated by protoc-gen-gogo. DO NOT EDIT.
// source: api.proto
/*
Package cri_runtimeoptions_v1 is a generated protocol buffer package.
It is generated from these files:
api.proto
It has these top-level messages:
Options
*/
package cri_runtimeoptions_v1
import proto "github.com/gogo/protobuf/proto"
import fmt "fmt"
import math "math"
import _ "github.com/gogo/protobuf/gogoproto"
import strings "strings"
import reflect "reflect"
import io "io"
// Reference imports to suppress errors if they are not otherwise used.
var _ = proto.Marshal
var _ = fmt.Errorf
var _ = math.Inf
// This is a compile-time assertion to ensure that this generated file
// is compatible with the proto package it is being compiled against.
// A compilation error at this line likely means your copy of the
// proto package needs to be updated.
const _ = proto.GoGoProtoPackageIsVersion2 // please upgrade the proto package
type Options struct {
// TypeUrl specifies the type of the content inside the config file.
TypeUrl string `protobuf:"bytes,1,opt,name=type_url,json=typeUrl,proto3" json:"type_url,omitempty"`
// ConfigPath specifies the filesystem location of the config file
// used by the runtime.
ConfigPath string `protobuf:"bytes,2,opt,name=config_path,json=configPath,proto3" json:"config_path,omitempty"`
}
func (m *Options) Reset() { *m = Options{} }
func (*Options) ProtoMessage() {}
func (*Options) Descriptor() ([]byte, []int) { return fileDescriptorApi, []int{0} }
func (m *Options) GetTypeUrl() string {
if m != nil {
return m.TypeUrl
}
return ""
}
func (m *Options) GetConfigPath() string {
if m != nil {
return m.ConfigPath
}
return ""
}
func init() {
proto.RegisterType((*Options)(nil), "cri.runtimeoptions.v1.Options")
}
func (m *Options) Marshal() (dAtA []byte, err error) {
size := m.Size()
dAtA = make([]byte, size)
n, err := m.MarshalTo(dAtA)
if err != nil {
return nil, err
}
return dAtA[:n], nil
}
func (m *Options) MarshalTo(dAtA []byte) (int, error) {
var i int
_ = i
var l int
_ = l
if len(m.TypeUrl) > 0 {
dAtA[i] = 0xa
i++
i = encodeVarintApi(dAtA, i, uint64(len(m.TypeUrl)))
i += copy(dAtA[i:], m.TypeUrl)
}
if len(m.ConfigPath) > 0 {
dAtA[i] = 0x12
i++
i = encodeVarintApi(dAtA, i, uint64(len(m.ConfigPath)))
i += copy(dAtA[i:], m.ConfigPath)
}
return i, nil
}
func encodeVarintApi(dAtA []byte, offset int, v uint64) int {
for v >= 1<<7 {
dAtA[offset] = uint8(v&0x7f | 0x80)
v >>= 7
offset++
}
dAtA[offset] = uint8(v)
return offset + 1
}
func (m *Options) Size() (n int) {
var l int
_ = l
l = len(m.TypeUrl)
if l > 0 {
n += 1 + l + sovApi(uint64(l))
}
l = len(m.ConfigPath)
if l > 0 {
n += 1 + l + sovApi(uint64(l))
}
return n
}
func sovApi(x uint64) (n int) {
for {
n++
x >>= 7
if x == 0 {
break
}
}
return n
}
func sozApi(x uint64) (n int) {
return sovApi(uint64((x << 1) ^ uint64((int64(x) >> 63))))
}
func (this *Options) String() string {
if this == nil {
return "nil"
}
s := strings.Join([]string{`&Options{`,
`TypeUrl:` + fmt.Sprintf("%v", this.TypeUrl) + `,`,
`ConfigPath:` + fmt.Sprintf("%v", this.ConfigPath) + `,`,
`}`,
}, "")
return s
}
func valueToStringApi(v interface{}) string {
rv := reflect.ValueOf(v)
if rv.IsNil() {
return "nil"
}
pv := reflect.Indirect(rv).Interface()
return fmt.Sprintf("*%v", pv)
}
func (m *Options) Unmarshal(dAtA []byte) error {
l := len(dAtA)
iNdEx := 0
for iNdEx < l {
preIndex := iNdEx
var wire uint64
for shift := uint(0); ; shift += 7 {
if shift >= 64 {
return ErrIntOverflowApi
}
if iNdEx >= l {
return io.ErrUnexpectedEOF
}
b := dAtA[iNdEx]
iNdEx++
wire |= (uint64(b) & 0x7F) << shift
if b < 0x80 {
break
}
}
fieldNum := int32(wire >> 3)
wireType := int(wire & 0x7)
if wireType == 4 {
return fmt.Errorf("proto: Options: wiretype end group for non-group")
}
if fieldNum <= 0 {
return fmt.Errorf("proto: Options: illegal tag %d (wire type %d)", fieldNum, wire)
}
switch fieldNum {
case 1:
if wireType != 2 {
return fmt.Errorf("proto: wrong wireType = %d for field TypeUrl", wireType)
}
var stringLen uint64
for shift := uint(0); ; shift += 7 {
if shift >= 64 {
return ErrIntOverflowApi
}
if iNdEx >= l {
return io.ErrUnexpectedEOF
}
b := dAtA[iNdEx]
iNdEx++
stringLen |= (uint64(b) & 0x7F) << shift
if b < 0x80 {
break
}
}
intStringLen := int(stringLen)
if intStringLen < 0 {
return ErrInvalidLengthApi
}
postIndex := iNdEx + intStringLen
if postIndex > l {
return io.ErrUnexpectedEOF
}
m.TypeUrl = string(dAtA[iNdEx:postIndex])
iNdEx = postIndex
case 2:
if wireType != 2 {
return fmt.Errorf("proto: wrong wireType = %d for field ConfigPath", wireType)
}
var stringLen uint64
for shift := uint(0); ; shift += 7 {
if shift >= 64 {
return ErrIntOverflowApi
}
if iNdEx >= l {
return io.ErrUnexpectedEOF
}
b := dAtA[iNdEx]
iNdEx++
stringLen |= (uint64(b) & 0x7F) << shift
if b < 0x80 {
break
}
}
intStringLen := int(stringLen)
if intStringLen < 0 {
return ErrInvalidLengthApi
}
postIndex := iNdEx + intStringLen
if postIndex > l {
return io.ErrUnexpectedEOF
}
m.ConfigPath = string(dAtA[iNdEx:postIndex])
iNdEx = postIndex
default:
iNdEx = preIndex
skippy, err := skipApi(dAtA[iNdEx:])
if err != nil {
return err
}
if skippy < 0 {
return ErrInvalidLengthApi
}
if (iNdEx + skippy) > l {
return io.ErrUnexpectedEOF
}
iNdEx += skippy
}
}
if iNdEx > l {
return io.ErrUnexpectedEOF
}
return nil
}
func skipApi(dAtA []byte) (n int, err error) {
l := len(dAtA)
iNdEx := 0
for iNdEx < l {
var wire uint64
for shift := uint(0); ; shift += 7 {
if shift >= 64 {
return 0, ErrIntOverflowApi
}
if iNdEx >= l {
return 0, io.ErrUnexpectedEOF
}
b := dAtA[iNdEx]
iNdEx++
wire |= (uint64(b) & 0x7F) << shift
if b < 0x80 {
break
}
}
wireType := int(wire & 0x7)
switch wireType {
case 0:
for shift := uint(0); ; shift += 7 {
if shift >= 64 {
return 0, ErrIntOverflowApi
}
if iNdEx >= l {
return 0, io.ErrUnexpectedEOF
}
iNdEx++
if dAtA[iNdEx-1] < 0x80 {
break
}
}
return iNdEx, nil
case 1:
iNdEx += 8
return iNdEx, nil
case 2:
var length int
for shift := uint(0); ; shift += 7 {
if shift >= 64 {
return 0, ErrIntOverflowApi
}
if iNdEx >= l {
return 0, io.ErrUnexpectedEOF
}
b := dAtA[iNdEx]
iNdEx++
length |= (int(b) & 0x7F) << shift
if b < 0x80 {
break
}
}
iNdEx += length
if length < 0 {
return 0, ErrInvalidLengthApi
}
return iNdEx, nil
case 3:
for {
var innerWire uint64
var start int = iNdEx
for shift := uint(0); ; shift += 7 {
if shift >= 64 {
return 0, ErrIntOverflowApi
}
if iNdEx >= l {
return 0, io.ErrUnexpectedEOF
}
b := dAtA[iNdEx]
iNdEx++
innerWire |= (uint64(b) & 0x7F) << shift
if b < 0x80 {
break
}
}
innerWireType := int(innerWire & 0x7)
if innerWireType == 4 {
break
}
next, err := skipApi(dAtA[start:])
if err != nil {
return 0, err
}
iNdEx = start + next
}
return iNdEx, nil
case 4:
return iNdEx, nil
case 5:
iNdEx += 4
return iNdEx, nil
default:
return 0, fmt.Errorf("proto: illegal wireType %d", wireType)
}
}
panic("unreachable")
}
var (
ErrInvalidLengthApi = fmt.Errorf("proto: negative length found during unmarshaling")
ErrIntOverflowApi = fmt.Errorf("proto: integer overflow")
)
func init() { proto.RegisterFile("api.proto", fileDescriptorApi) }
var fileDescriptorApi = []byte{
// 183 bytes of a gzipped FileDescriptorProto
0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xe2, 0xe2, 0x4c, 0x2c, 0xc8, 0xd4,
0x2b, 0x28, 0xca, 0x2f, 0xc9, 0x17, 0x12, 0x4d, 0x2e, 0xca, 0xd4, 0x2b, 0x2a, 0xcd, 0x2b, 0xc9,
0xcc, 0x4d, 0xcd, 0x2f, 0x28, 0xc9, 0xcc, 0xcf, 0x2b, 0xd6, 0x2b, 0x33, 0x94, 0xd2, 0x4d, 0xcf,
0x2c, 0xc9, 0x28, 0x4d, 0xd2, 0x4b, 0xce, 0xcf, 0xd5, 0x4f, 0xcf, 0x4f, 0xcf, 0xd7, 0x07, 0xab,
0x4e, 0x2a, 0x4d, 0x03, 0xf3, 0xc0, 0x1c, 0x30, 0x0b, 0x62, 0x8a, 0x92, 0x2b, 0x17, 0xbb, 0x3f,
0x44, 0xb3, 0x90, 0x24, 0x17, 0x47, 0x49, 0x65, 0x41, 0x6a, 0x7c, 0x69, 0x51, 0x8e, 0x04, 0xa3,
0x02, 0xa3, 0x06, 0x67, 0x10, 0x3b, 0x88, 0x1f, 0x5a, 0x94, 0x23, 0x24, 0xcf, 0xc5, 0x9d, 0x9c,
0x9f, 0x97, 0x96, 0x99, 0x1e, 0x5f, 0x90, 0x58, 0x92, 0x21, 0xc1, 0x04, 0x96, 0xe5, 0x82, 0x08,
0x05, 0x24, 0x96, 0x64, 0x38, 0xc9, 0x9c, 0x78, 0x28, 0xc7, 0x78, 0xe3, 0xa1, 0x1c, 0x43, 0xc3,
0x23, 0x39, 0xc6, 0x13, 0x8f, 0xe4, 0x18, 0x2f, 0x3c, 0x92, 0x63, 0x7c, 0xf0, 0x48, 0x8e, 0x71,
0xc2, 0x63, 0x39, 0x86, 0x24, 0x36, 0xb0, 0x5d, 0xc6, 0x80, 0x00, 0x00, 0x00, 0xff, 0xff, 0x07,
0x00, 0xf2, 0x18, 0xbe, 0x00, 0x00, 0x00,
}

22
pkg/api/runtimeoptions/v1/api.proto Normal file
View File

@ -0,0 +1,22 @@
// To regenerate api.pb.go run `make proto`
syntax = "proto3";
package cri.runtimeoptions.v1;
import "github.com/gogo/protobuf/gogoproto/gogo.proto";
option (gogoproto.goproto_stringer_all) = false;
option (gogoproto.stringer_all) = true;
option (gogoproto.goproto_getters_all) = true;
option (gogoproto.marshaler_all) = true;
option (gogoproto.sizer_all) = true;
option (gogoproto.unmarshaler_all) = true;
option (gogoproto.goproto_unrecognized_all) = false;
message Options {
// TypeUrl specifies the type of the content inside the config file.
string type_url = 1;
// ConfigPath specifies the filesystem location of the config file
// used by the runtime.
string config_path = 2;
}

54
pkg/atomic/atomic_boolean.go Normal file
View File

@ -0,0 +1,54 @@
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package atomic
import "sync/atomic"
// Bool is an atomic Boolean,
// Its methods are all atomic, thus safe to be called by
// multiple goroutines simultaneously.
type Bool interface {
Set()
Unset()
IsSet() bool
}
// NewBool creates an Bool with given default value
func NewBool(ok bool) Bool {
ab := new(atomicBool)
if ok {
ab.Set()
}
return ab
}
type atomicBool int32
// Set sets the Boolean to true
func (ab *atomicBool) Set() {
atomic.StoreInt32((*int32)(ab), 1)
}
// Unset sets the Boolean to false
func (ab *atomicBool) Unset() {
atomic.StoreInt32((*int32)(ab), 0)
}
// IsSet returns whether the Boolean is true
func (ab *atomicBool) IsSet() bool {
return atomic.LoadInt32((*int32)(ab)) == 1
}

32
pkg/atomic/atomic_boolean_test.go Normal file
View File

@ -0,0 +1,32 @@
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package atomic
import (
"testing"
"github.com/stretchr/testify/assert"
)
func TestBoolean(t *testing.T) {
ab := NewBool(true)
assert.True(t, ab.IsSet())
ab.Unset()
assert.False(t, ab.IsSet())
ab.Set()
assert.True(t, ab.IsSet())
}

369
pkg/config/config.go Normal file
View File

@ -0,0 +1,369 @@
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package config
import (
"context"
"time"
"github.com/BurntSushi/toml"
"github.com/containerd/containerd/log"
"github.com/containerd/containerd/plugin"
"github.com/pkg/errors"
)
// Runtime struct to contain the type(ID), engine, and root variables for a default runtime
// and a runtime for untrusted worload.
type Runtime struct {
// Type is the runtime type to use in containerd e.g. io.containerd.runtime.v1.linux
Type string `toml:"runtime_type" json:"runtimeType"`
// Engine is the name of the runtime engine used by containerd.
// This only works for runtime type "io.containerd.runtime.v1.linux".
// DEPRECATED: use Options instead. Remove when shim v1 is deprecated.
Engine string `toml:"runtime_engine" json:"runtimeEngine"`
// PodAnnotations is a list of pod annotations passed to both pod sandbox as well as
// container OCI annotations.
PodAnnotations []string `toml:"pod_annotations" json:"PodAnnotations"`
// ContainerAnnotations is a list of container annotations passed through to the OCI config of the containers.
// Container annotations in CRI are usually generated by other Kubernetes node components (i.e., not users).
// Currently, only device plugins populate the annotations.
ContainerAnnotations []string `toml:"container_annotations" json:"ContainerAnnotations"`
// Root is the directory used by containerd for runtime state.
// DEPRECATED: use Options instead. Remove when shim v1 is deprecated.
// This only works for runtime type "io.containerd.runtime.v1.linux".
Root string `toml:"runtime_root" json:"runtimeRoot"`
// Options are config options for the runtime. If options is loaded
// from toml config, it will be toml.Primitive.
Options *toml.Primitive `toml:"options" json:"options"`
// PrivilegedWithoutHostDevices overloads the default behaviour for adding host devices to the
// runtime spec when the container is privileged. Defaults to false.
PrivilegedWithoutHostDevices bool `toml:"privileged_without_host_devices" json:"privileged_without_host_devices"`
// BaseRuntimeSpec is a json file with OCI spec to use as base spec that all container's will be created from.
BaseRuntimeSpec string `toml:"base_runtime_spec" json:"baseRuntimeSpec"`
}
// ContainerdConfig contains toml config related to containerd
type ContainerdConfig struct {
// Snapshotter is the snapshotter used by containerd.
Snapshotter string `toml:"snapshotter" json:"snapshotter"`
// DefaultRuntimeName is the default runtime name to use from the runtimes table.
DefaultRuntimeName string `toml:"default_runtime_name" json:"defaultRuntimeName"`
// DefaultRuntime is the default runtime to use in containerd.
// This runtime is used when no runtime handler (or the empty string) is provided.
// DEPRECATED: use DefaultRuntimeName instead. Remove in containerd 1.4.
DefaultRuntime Runtime `toml:"default_runtime" json:"defaultRuntime"`
// UntrustedWorkloadRuntime is a runtime to run untrusted workloads on it.
// DEPRECATED: use `untrusted` runtime in Runtimes instead. Remove in containerd 1.4.
UntrustedWorkloadRuntime Runtime `toml:"untrusted_workload_runtime" json:"untrustedWorkloadRuntime"`
// Runtimes is a map from CRI RuntimeHandler strings, which specify types of runtime
// configurations, to the matching configurations.
Runtimes map[string]Runtime `toml:"runtimes" json:"runtimes"`
// NoPivot disables pivot-root (linux only), required when running a container in a RamDisk with runc
// This only works for runtime type "io.containerd.runtime.v1.linux".
NoPivot bool `toml:"no_pivot" json:"noPivot"`
// DisableSnapshotAnnotations disables to pass additional annotations (image
// related information) to snapshotters. These annotations are required by
// stargz snapshotter (https://github.com/containerd/stargz-snapshotter).
DisableSnapshotAnnotations bool `toml:"disable_snapshot_annotations" json:"disableSnapshotAnnotations"`
// DiscardUnpackedLayers is a boolean flag to specify whether to allow GC to
// remove layers from the content store after successfully unpacking these
// layers to the snapshotter.
DiscardUnpackedLayers bool `toml:"discard_unpacked_layers" json:"discardUnpackedLayers"`
}
// CniConfig contains toml config related to cni
type CniConfig struct {
// NetworkPluginBinDir is the directory in which the binaries for the plugin is kept.
NetworkPluginBinDir string `toml:"bin_dir" json:"binDir"`
// NetworkPluginConfDir is the directory in which the admin places a CNI conf.
NetworkPluginConfDir string `toml:"conf_dir" json:"confDir"`
// NetworkPluginMaxConfNum is the max number of plugin config files that will
// be loaded from the cni config directory by go-cni. Set the value to 0 to
// load all config files (no arbitrary limit). The legacy default value is 1.
NetworkPluginMaxConfNum int `toml:"max_conf_num" json:"maxConfNum"`
// NetworkPluginConfTemplate is the file path of golang template used to generate
// cni config.
// When it is set, containerd will get cidr(s) from kubelet to replace {{.PodCIDR}},
// {{.PodCIDRRanges}} or {{.Routes}} in the template, and write the config into
// NetworkPluginConfDir.
// Ideally the cni config should be placed by system admin or cni daemon like calico,
// weaveworks etc. However, there are still users using kubenet
// (https://kubernetes.io/docs/concepts/cluster-administration/network-plugins/#kubenet)
// today, who don't have a cni daemonset in production. NetworkPluginConfTemplate is
// a temporary backward-compatible solution for them.
// TODO(random-liu): Deprecate this option when kubenet is deprecated.
NetworkPluginConfTemplate string `toml:"conf_template" json:"confTemplate"`
}
// Mirror contains the config related to the registry mirror
type Mirror struct {
// Endpoints are endpoints for a namespace. CRI plugin will try the endpoints
// one by one until a working one is found. The endpoint must be a valid url
// with host specified.
// The scheme, host and path from the endpoint URL will be used.
Endpoints []string `toml:"endpoint" json:"endpoint"`
}
// AuthConfig contains the config related to authentication to a specific registry
type AuthConfig struct {
// Username is the username to login the registry.
Username string `toml:"username" json:"username"`
// Password is the password to login the registry.
Password string `toml:"password" json:"password"`
// Auth is a base64 encoded string from the concatenation of the username,
// a colon, and the password.
Auth string `toml:"auth" json:"auth"`
// IdentityToken is used to authenticate the user and get
// an access token for the registry.
IdentityToken string `toml:"identitytoken" json:"identitytoken"`
}
// TLSConfig contains the CA/Cert/Key used for a registry
type TLSConfig struct {
InsecureSkipVerify bool `toml:"insecure_skip_verify" json:"insecure_skip_verify"`
CAFile string `toml:"ca_file" json:"caFile"`
CertFile string `toml:"cert_file" json:"certFile"`
KeyFile string `toml:"key_file" json:"keyFile"`
}
// Registry is registry settings configured
type Registry struct {
// Mirrors are namespace to mirror mapping for all namespaces.
Mirrors map[string]Mirror `toml:"mirrors" json:"mirrors"`
// Configs are configs for each registry.
// The key is the domain name or IP of the registry.
Configs map[string]RegistryConfig `toml:"configs" json:"configs"`
// Auths are registry endpoint to auth config mapping. The registry endpoint must
// be a valid url with host specified.
// DEPRECATED: Use Configs instead. Remove in containerd 1.4.
Auths map[string]AuthConfig `toml:"auths" json:"auths"`
// Headers adds additional HTTP headers that get sent to all registries
Headers map[string][]string `toml:"headers" json:"headers"`
}
// RegistryConfig contains configuration used to communicate with the registry.
type RegistryConfig struct {
// Auth contains information to authenticate to the registry.
Auth *AuthConfig `toml:"auth" json:"auth"`
// TLS is a pair of CA/Cert/Key which then are used when creating the transport
// that communicates with the registry.
TLS *TLSConfig `toml:"tls" json:"tls"`
}
// ImageDecryption contains configuration to handling decryption of encrypted container images.
type ImageDecryption struct {
// KeyModel specifies the trust model of where keys should reside.
//
// Details of field usage can be found in:
// https://github.com/containerd/cri/tree/master/docs/config.md
//
// Details of key models can be found in:
// https://github.com/containerd/cri/tree/master/docs/decryption.md
KeyModel string `toml:"key_model" json:"keyModel"`
}
// PluginConfig contains toml config related to CRI plugin,
// it is a subset of Config.
type PluginConfig struct {
// ContainerdConfig contains config related to containerd
ContainerdConfig `toml:"containerd" json:"containerd"`
// CniConfig contains config related to cni
CniConfig `toml:"cni" json:"cni"`
// Registry contains config related to the registry
Registry Registry `toml:"registry" json:"registry"`
// ImageDecryption contains config related to handling decryption of encrypted container images
ImageDecryption `toml:"image_decryption" json:"imageDecryption"`
// DisableTCPService disables serving CRI on the TCP server.
DisableTCPService bool `toml:"disable_tcp_service" json:"disableTCPService"`
// StreamServerAddress is the ip address streaming server is listening on.
StreamServerAddress string `toml:"stream_server_address" json:"streamServerAddress"`
// StreamServerPort is the port streaming server is listening on.
StreamServerPort string `toml:"stream_server_port" json:"streamServerPort"`
// StreamIdleTimeout is the maximum time a streaming connection
// can be idle before the connection is automatically closed.
// The string is in the golang duration format, see:
// https://golang.org/pkg/time/#ParseDuration
StreamIdleTimeout string `toml:"stream_idle_timeout" json:"streamIdleTimeout"`
// EnableSelinux indicates to enable the selinux support.
EnableSelinux bool `toml:"enable_selinux" json:"enableSelinux"`
// SelinuxCategoryRange allows the upper bound on the category range to be set.
// If not specified or set to 0, defaults to 1024 from the selinux package.
SelinuxCategoryRange int `toml:"selinux_category_range" json:"selinuxCategoryRange"`
// SandboxImage is the image used by sandbox container.
SandboxImage string `toml:"sandbox_image" json:"sandboxImage"`
// StatsCollectPeriod is the period (in seconds) of snapshots stats collection.
StatsCollectPeriod int `toml:"stats_collect_period" json:"statsCollectPeriod"`
// SystemdCgroup enables systemd cgroup support.
// This only works for runtime type "io.containerd.runtime.v1.linux".
// DEPRECATED: config runc runtime handler instead. Remove when shim v1 is deprecated.
SystemdCgroup bool `toml:"systemd_cgroup" json:"systemdCgroup"`
// EnableTLSStreaming indicates to enable the TLS streaming support.
EnableTLSStreaming bool `toml:"enable_tls_streaming" json:"enableTLSStreaming"`
// X509KeyPairStreaming is a x509 key pair used for TLS streaming
X509KeyPairStreaming `toml:"x509_key_pair_streaming" json:"x509KeyPairStreaming"`
// MaxContainerLogLineSize is the maximum log line size in bytes for a container.
// Log line longer than the limit will be split into multiple lines. Non-positive
// value means no limit.
MaxContainerLogLineSize int `toml:"max_container_log_line_size" json:"maxContainerLogSize"`
// DisableCgroup indicates to disable the cgroup support.
// This is useful when the containerd does not have permission to access cgroup.
DisableCgroup bool `toml:"disable_cgroup" json:"disableCgroup"`
// DisableApparmor indicates to disable the apparmor support.
// This is useful when the containerd does not have permission to access Apparmor.
DisableApparmor bool `toml:"disable_apparmor" json:"disableApparmor"`
// RestrictOOMScoreAdj indicates to limit the lower bound of OOMScoreAdj to the containerd's
// current OOMScoreADj.
// This is useful when the containerd does not have permission to decrease OOMScoreAdj.
RestrictOOMScoreAdj bool `toml:"restrict_oom_score_adj" json:"restrictOOMScoreAdj"`
// MaxConcurrentDownloads restricts the number of concurrent downloads for each image.
MaxConcurrentDownloads int `toml:"max_concurrent_downloads" json:"maxConcurrentDownloads"`
// DisableProcMount disables Kubernetes ProcMount support. This MUST be set to `true`
// when using containerd with Kubernetes <=1.11.
DisableProcMount bool `toml:"disable_proc_mount" json:"disableProcMount"`
// UnsetSeccompProfile is the profile containerd/cri will use If the provided seccomp profile is
// unset (`""`) for a container (default is `unconfined`)
UnsetSeccompProfile string `toml:"unset_seccomp_profile" json:"unsetSeccompProfile"`
// TolerateMissingHugetlbController if set to false will error out on create/update
// container requests with huge page limits if the cgroup controller for hugepages is not present.
// This helps with supporting Kubernetes <=1.18 out of the box. (default is `true`)
TolerateMissingHugetlbController bool `toml:"tolerate_missing_hugetlb_controller" json:"tolerateMissingHugetlbController"`
// DisableHugetlbController indicates to silently disable the hugetlb controller, even when it is
// present in /sys/fs/cgroup/cgroup.controllers.
// This helps with running rootless mode + cgroup v2 + systemd but without hugetlb delegation.
DisableHugetlbController bool `toml:"disable_hugetlb_controller" json:"disableHugetlbController"`
// IgnoreImageDefinedVolumes ignores volumes defined by the image. Useful for better resource
// isolation, security and early detection of issues in the mount configuration when using
// ReadOnlyRootFilesystem since containers won't silently mount a temporary volume.
IgnoreImageDefinedVolumes bool `toml:"ignore_image_defined_volumes" json:"ignoreImageDefinedVolumes"`
}
// X509KeyPairStreaming contains the x509 configuration for streaming
type X509KeyPairStreaming struct {
// TLSCertFile is the path to a certificate file
TLSCertFile string `toml:"tls_cert_file" json:"tlsCertFile"`
// TLSKeyFile is the path to a private key file
TLSKeyFile string `toml:"tls_key_file" json:"tlsKeyFile"`
}
// Config contains all configurations for cri server.
type Config struct {
// PluginConfig is the config for CRI plugin.
PluginConfig
// ContainerdRootDir is the root directory path for containerd.
ContainerdRootDir string `json:"containerdRootDir"`
// ContainerdEndpoint is the containerd endpoint path.
ContainerdEndpoint string `json:"containerdEndpoint"`
// RootDir is the root directory path for managing cri plugin files
// (metadata checkpoint etc.)
RootDir string `json:"rootDir"`
// StateDir is the root directory path for managing volatile pod/container data
StateDir string `json:"stateDir"`
}
const (
// RuntimeUntrusted is the implicit runtime defined for ContainerdConfig.UntrustedWorkloadRuntime
RuntimeUntrusted = "untrusted"
// RuntimeDefault is the implicit runtime defined for ContainerdConfig.DefaultRuntime
RuntimeDefault = "default"
// KeyModelNode is the key model where key for encrypted images reside
// on the worker nodes
KeyModelNode = "node"
)
// ValidatePluginConfig validates the given plugin configuration.
func ValidatePluginConfig(ctx context.Context, c *PluginConfig) error {
if c.ContainerdConfig.Runtimes == nil {
c.ContainerdConfig.Runtimes = make(map[string]Runtime)
}
// Validation for deprecated untrusted_workload_runtime.
if c.ContainerdConfig.UntrustedWorkloadRuntime.Type != "" {
log.G(ctx).Warning("`untrusted_workload_runtime` is deprecated, please use `untrusted` runtime in `runtimes` instead")
if _, ok := c.ContainerdConfig.Runtimes[RuntimeUntrusted]; ok {
return errors.Errorf("conflicting definitions: configuration includes both `untrusted_workload_runtime` and `runtimes[%q]`", RuntimeUntrusted)
}
c.ContainerdConfig.Runtimes[RuntimeUntrusted] = c.ContainerdConfig.UntrustedWorkloadRuntime
}
// Validation for deprecated default_runtime field.
if c.ContainerdConfig.DefaultRuntime.Type != "" {
log.G(ctx).Warning("`default_runtime` is deprecated, please use `default_runtime_name` to reference the default configuration you have defined in `runtimes`")
c.ContainerdConfig.DefaultRuntimeName = RuntimeDefault
c.ContainerdConfig.Runtimes[RuntimeDefault] = c.ContainerdConfig.DefaultRuntime
}
// Validation for default_runtime_name
if c.ContainerdConfig.DefaultRuntimeName == "" {
return errors.New("`default_runtime_name` is empty")
}
if _, ok := c.ContainerdConfig.Runtimes[c.ContainerdConfig.DefaultRuntimeName]; !ok {
return errors.New("no corresponding runtime configured in `runtimes` for `default_runtime_name`")
}
// Validation for deprecated runtime options.
if c.SystemdCgroup {
if c.ContainerdConfig.Runtimes[c.ContainerdConfig.DefaultRuntimeName].Type != plugin.RuntimeLinuxV1 {
return errors.Errorf("`systemd_cgroup` only works for runtime %s", plugin.RuntimeLinuxV1)
}
log.G(ctx).Warning("`systemd_cgroup` is deprecated, please use runtime `options` instead")
}
if c.NoPivot {
if c.ContainerdConfig.Runtimes[c.ContainerdConfig.DefaultRuntimeName].Type != plugin.RuntimeLinuxV1 {
return errors.Errorf("`no_pivot` only works for runtime %s", plugin.RuntimeLinuxV1)
}
// NoPivot can't be deprecated yet, because there is no alternative config option
// for `io.containerd.runtime.v1.linux`.
}
for _, r := range c.ContainerdConfig.Runtimes {
if r.Engine != "" {
if r.Type != plugin.RuntimeLinuxV1 {
return errors.Errorf("`runtime_engine` only works for runtime %s", plugin.RuntimeLinuxV1)
}
log.G(ctx).Warning("`runtime_engine` is deprecated, please use runtime `options` instead")
}
if r.Root != "" {
if r.Type != plugin.RuntimeLinuxV1 {
return errors.Errorf("`runtime_root` only works for runtime %s", plugin.RuntimeLinuxV1)
}
log.G(ctx).Warning("`runtime_root` is deprecated, please use runtime `options` instead")
}
}
// Validation for deprecated auths options and mapping it to configs.
if len(c.Registry.Auths) != 0 {
if c.Registry.Configs == nil {
c.Registry.Configs = make(map[string]RegistryConfig)
}
for endpoint, auth := range c.Registry.Auths {
config := c.Registry.Configs[endpoint]
config.Auth = &auth
c.Registry.Configs[endpoint] = config
}
log.G(ctx).Warning("`auths` is deprecated, please use registry`configs` instead")
}
// Validation for stream_idle_timeout
if c.StreamIdleTimeout != "" {
if _, err := time.ParseDuration(c.StreamIdleTimeout); err != nil {
return errors.Wrap(err, "invalid stream idle timeout")
}
}
return nil
}

334
pkg/config/config_test.go Normal file
View File

@ -0,0 +1,334 @@
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package config
import (
"context"
"fmt"
"testing"
"github.com/containerd/containerd/plugin"
"github.com/stretchr/testify/assert"
)
func TestValidateConfig(t *testing.T) {
for desc, test := range map[string]struct {
config *PluginConfig
expectedErr string
expected *PluginConfig
}{
"deprecated untrusted_workload_runtime": {
config: &PluginConfig{
ContainerdConfig: ContainerdConfig{
DefaultRuntimeName: RuntimeDefault,
UntrustedWorkloadRuntime: Runtime{
Type: "untrusted",
},
Runtimes: map[string]Runtime{
RuntimeDefault: {
Type: "default",
},
},
},
},
expected: &PluginConfig{
ContainerdConfig: ContainerdConfig{
DefaultRuntimeName: RuntimeDefault,
UntrustedWorkloadRuntime: Runtime{
Type: "untrusted",
},
Runtimes: map[string]Runtime{
RuntimeUntrusted: {
Type: "untrusted",
},
RuntimeDefault: {
Type: "default",
},
},
},
},
},
"both untrusted_workload_runtime and runtime[untrusted]": {
config: &PluginConfig{
ContainerdConfig: ContainerdConfig{
DefaultRuntimeName: RuntimeDefault,
UntrustedWorkloadRuntime: Runtime{
Type: "untrusted-1",
},
Runtimes: map[string]Runtime{
RuntimeUntrusted: {
Type: "untrusted-2",
},
RuntimeDefault: {
Type: "default",
},
},
},
},
expectedErr: fmt.Sprintf("conflicting definitions: configuration includes both `untrusted_workload_runtime` and `runtimes[%q]`", RuntimeUntrusted),
},
"deprecated default_runtime": {
config: &PluginConfig{
ContainerdConfig: ContainerdConfig{
DefaultRuntime: Runtime{
Type: "default",
},
},
},
expected: &PluginConfig{
ContainerdConfig: ContainerdConfig{
DefaultRuntime: Runtime{
Type: "default",
},
DefaultRuntimeName: RuntimeDefault,
Runtimes: map[string]Runtime{
RuntimeDefault: {
Type: "default",
},
},
},
},
},
"no default_runtime_name": {
config: &PluginConfig{},
expectedErr: "`default_runtime_name` is empty",
},
"no runtime[default_runtime_name]": {
config: &PluginConfig{
ContainerdConfig: ContainerdConfig{
DefaultRuntimeName: RuntimeDefault,
},
},
expectedErr: "no corresponding runtime configured in `runtimes` for `default_runtime_name`",
},
"deprecated systemd_cgroup for v1 runtime": {
config: &PluginConfig{
SystemdCgroup: true,
ContainerdConfig: ContainerdConfig{
DefaultRuntimeName: RuntimeDefault,
Runtimes: map[string]Runtime{
RuntimeDefault: {
Type: plugin.RuntimeLinuxV1,
},
},
},
},
expected: &PluginConfig{
SystemdCgroup: true,
ContainerdConfig: ContainerdConfig{
DefaultRuntimeName: RuntimeDefault,
Runtimes: map[string]Runtime{
RuntimeDefault: {
Type: plugin.RuntimeLinuxV1,
},
},
},
},
},
"deprecated systemd_cgroup for v2 runtime": {
config: &PluginConfig{
SystemdCgroup: true,
ContainerdConfig: ContainerdConfig{
DefaultRuntimeName: RuntimeDefault,
Runtimes: map[string]Runtime{
RuntimeDefault: {
Type: plugin.RuntimeRuncV1,
},
},
},
},
expectedErr: fmt.Sprintf("`systemd_cgroup` only works for runtime %s", plugin.RuntimeLinuxV1),
},
"no_pivot for v1 runtime": {
config: &PluginConfig{
ContainerdConfig: ContainerdConfig{
NoPivot: true,
DefaultRuntimeName: RuntimeDefault,
Runtimes: map[string]Runtime{
RuntimeDefault: {
Type: plugin.RuntimeLinuxV1,
},
},
},
},
expected: &PluginConfig{
ContainerdConfig: ContainerdConfig{
NoPivot: true,
DefaultRuntimeName: RuntimeDefault,
Runtimes: map[string]Runtime{
RuntimeDefault: {
Type: plugin.RuntimeLinuxV1,
},
},
},
},
},
"no_pivot for v2 runtime": {
config: &PluginConfig{
ContainerdConfig: ContainerdConfig{
NoPivot: true,
DefaultRuntimeName: RuntimeDefault,
Runtimes: map[string]Runtime{
RuntimeDefault: {
Type: plugin.RuntimeRuncV1,
},
},
},
},
expectedErr: fmt.Sprintf("`no_pivot` only works for runtime %s", plugin.RuntimeLinuxV1),
},
"deprecated runtime_engine for v1 runtime": {
config: &PluginConfig{
ContainerdConfig: ContainerdConfig{
DefaultRuntimeName: RuntimeDefault,
Runtimes: map[string]Runtime{
RuntimeDefault: {
Engine: "runc",
Type: plugin.RuntimeLinuxV1,
},
},
},
},
expected: &PluginConfig{
ContainerdConfig: ContainerdConfig{
DefaultRuntimeName: RuntimeDefault,
Runtimes: map[string]Runtime{
RuntimeDefault: {
Engine: "runc",
Type: plugin.RuntimeLinuxV1,
},
},
},
},
},
"deprecated runtime_engine for v2 runtime": {
config: &PluginConfig{
ContainerdConfig: ContainerdConfig{
DefaultRuntimeName: RuntimeDefault,
Runtimes: map[string]Runtime{
RuntimeDefault: {
Engine: "runc",
Type: plugin.RuntimeRuncV1,
},
},
},
},
expectedErr: fmt.Sprintf("`runtime_engine` only works for runtime %s", plugin.RuntimeLinuxV1),
},
"deprecated runtime_root for v1 runtime": {
config: &PluginConfig{
ContainerdConfig: ContainerdConfig{
DefaultRuntimeName: RuntimeDefault,
Runtimes: map[string]Runtime{
RuntimeDefault: {
Root: "/run/containerd/runc",
Type: plugin.RuntimeLinuxV1,
},
},
},
},
expected: &PluginConfig{
ContainerdConfig: ContainerdConfig{
DefaultRuntimeName: RuntimeDefault,
Runtimes: map[string]Runtime{
RuntimeDefault: {
Root: "/run/containerd/runc",
Type: plugin.RuntimeLinuxV1,
},
},
},
},
},
"deprecated runtime_root for v2 runtime": {
config: &PluginConfig{
ContainerdConfig: ContainerdConfig{
DefaultRuntimeName: RuntimeDefault,
Runtimes: map[string]Runtime{
RuntimeDefault: {
Root: "/run/containerd/runc",
Type: plugin.RuntimeRuncV1,
},
},
},
},
expectedErr: fmt.Sprintf("`runtime_root` only works for runtime %s", plugin.RuntimeLinuxV1),
},
"deprecated auths": {
config: &PluginConfig{
ContainerdConfig: ContainerdConfig{
DefaultRuntimeName: RuntimeDefault,
Runtimes: map[string]Runtime{
RuntimeDefault: {
Type: plugin.RuntimeRuncV1,
},
},
},
Registry: Registry{
Auths: map[string]AuthConfig{
"https://gcr.io": {Username: "test"},
},
},
},
expected: &PluginConfig{
ContainerdConfig: ContainerdConfig{
DefaultRuntimeName: RuntimeDefault,
Runtimes: map[string]Runtime{
RuntimeDefault: {
Type: plugin.RuntimeRuncV1,
},
},
},
Registry: Registry{
Configs: map[string]RegistryConfig{
"https://gcr.io": {
Auth: &AuthConfig{
Username: "test",
},
},
},
Auths: map[string]AuthConfig{
"https://gcr.io": {Username: "test"},
},
},
},
},
"invalid stream_idle_timeout": {
config: &PluginConfig{
StreamIdleTimeout: "invalid",
ContainerdConfig: ContainerdConfig{
DefaultRuntimeName: RuntimeDefault,
Runtimes: map[string]Runtime{
RuntimeDefault: {
Type: "default",
},
},
},
},
expectedErr: "invalid stream idle timeout",
},
} {
t.Run(desc, func(t *testing.T) {
err := ValidatePluginConfig(context.Background(), test.config)
if test.expectedErr != "" {
assert.Contains(t, err.Error(), test.expectedErr)
} else {
assert.NoError(t, err)
assert.Equal(t, test.expected, test.config)
}
})
}
}

75
pkg/config/config_unix.go Normal file
View File

@ -0,0 +1,75 @@
// +build !windows
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package config
import (
"github.com/BurntSushi/toml"
"github.com/containerd/containerd"
"github.com/containerd/cri/pkg/streaming"
)
// DefaultConfig returns default configurations of cri plugin.
func DefaultConfig() PluginConfig {
return PluginConfig{
CniConfig: CniConfig{
NetworkPluginBinDir: "/opt/cni/bin",
NetworkPluginConfDir: "/etc/cni/net.d",
NetworkPluginMaxConfNum: 1, // only one CNI plugin config file will be loaded
NetworkPluginConfTemplate: "",
},
ContainerdConfig: ContainerdConfig{
Snapshotter: containerd.DefaultSnapshotter,
DefaultRuntimeName: "runc",
NoPivot: false,
Runtimes: map[string]Runtime{
"runc": {
Type: "io.containerd.runc.v2",
Options: new(toml.Primitive),
},
},
},
DisableTCPService: true,
StreamServerAddress: "127.0.0.1",
StreamServerPort: "0",
StreamIdleTimeout: streaming.DefaultConfig.StreamIdleTimeout.String(), // 4 hour
EnableSelinux: false,
SelinuxCategoryRange: 1024,
EnableTLSStreaming: false,
X509KeyPairStreaming: X509KeyPairStreaming{
TLSKeyFile: "",
TLSCertFile: "",
},
SandboxImage: "k8s.gcr.io/pause:3.2",
StatsCollectPeriod: 10,
SystemdCgroup: false,
MaxContainerLogLineSize: 16 * 1024,
Registry: Registry{
Mirrors: map[string]Mirror{
"docker.io": {
Endpoints: []string{"https://registry-1.docker.io"},
},
},
},
MaxConcurrentDownloads: 3,
DisableProcMount: false,
TolerateMissingHugetlbController: true,
DisableHugetlbController: true,
IgnoreImageDefinedVolumes: false,
}
}

71
pkg/config/config_windows.go Normal file
View File

@ -0,0 +1,71 @@
// +build windows
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package config
import (
"os"
"path/filepath"
"github.com/containerd/containerd"
"github.com/containerd/cri/pkg/streaming"
)
// DefaultConfig returns default configurations of cri plugin.
func DefaultConfig() PluginConfig {
return PluginConfig{
CniConfig: CniConfig{
NetworkPluginBinDir: filepath.Join(os.Getenv("ProgramFiles"), "containerd", "cni", "bin"),
NetworkPluginConfDir: filepath.Join(os.Getenv("ProgramFiles"), "containerd", "cni", "conf"),
NetworkPluginMaxConfNum: 1,
NetworkPluginConfTemplate: "",
},
ContainerdConfig: ContainerdConfig{
Snapshotter: containerd.DefaultSnapshotter,
DefaultRuntimeName: "runhcs-wcow-process",
NoPivot: false,
Runtimes: map[string]Runtime{
"runhcs-wcow-process": {
Type: "io.containerd.runhcs.v1",
},
},
},
DisableTCPService: true,
StreamServerAddress: "127.0.0.1",
StreamServerPort: "0",
StreamIdleTimeout: streaming.DefaultConfig.StreamIdleTimeout.String(), // 4 hour
EnableTLSStreaming: false,
X509KeyPairStreaming: X509KeyPairStreaming{
TLSKeyFile: "",
TLSCertFile: "",
},
SandboxImage: "mcr.microsoft.com/oss/kubernetes/pause:1.4.0",
StatsCollectPeriod: 10,
MaxContainerLogLineSize: 16 * 1024,
Registry: Registry{
Mirrors: map[string]Mirror{
"docker.io": {
Endpoints: []string{"https://registry-1.docker.io"},
},
},
},
MaxConcurrentDownloads: 3,
IgnoreImageDefinedVolumes: false,
// TODO(windows): Add platform specific config, so that most common defaults can be shared.
}
}

26
pkg/constants/constants.go Normal file
View File

@ -0,0 +1,26 @@
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package constants
// TODO(random-liu): Merge annotations package into this package.
const (
// K8sContainerdNamespace is the namespace we use to connect containerd.
K8sContainerdNamespace = "k8s.io"
// CRIVersion is the CRI version supported by the CRI plugin.
CRIVersion = "v1alpha2"
)

118
pkg/containerd/opts/container.go Normal file
View File

@ -0,0 +1,118 @@
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package opts
import (
"context"
"io/ioutil"
"os"
"path/filepath"
"github.com/containerd/containerd"
"github.com/containerd/containerd/containers"
"github.com/containerd/containerd/errdefs"
"github.com/containerd/containerd/log"
"github.com/containerd/containerd/mount"
"github.com/containerd/continuity/fs"
"github.com/pkg/errors"
)
// WithNewSnapshot wraps `containerd.WithNewSnapshot` so that if creating the
// snapshot fails we make sure the image is actually unpacked and and retry.
func WithNewSnapshot(id string, i containerd.Image) containerd.NewContainerOpts {
f := containerd.WithNewSnapshot(id, i)
return func(ctx context.Context, client *containerd.Client, c *containers.Container) error {
if err := f(ctx, client, c); err != nil {
if !errdefs.IsNotFound(err) {
return err
}
if err := i.Unpack(ctx, c.Snapshotter); err != nil {
return errors.Wrap(err, "error unpacking image")
}
return f(ctx, client, c)
}
return nil
}
}
// WithVolumes copies ownership of volume in rootfs to its corresponding host path.
// It doesn't update runtime spec.
// The passed in map is a host path to container path map for all volumes.
func WithVolumes(volumeMounts map[string]string) containerd.NewContainerOpts {
return func(ctx context.Context, client *containerd.Client, c *containers.Container) (err error) {
if c.Snapshotter == "" {
return errors.New("no snapshotter set for container")
}
if c.SnapshotKey == "" {
return errors.New("rootfs not created for container")
}
snapshotter := client.SnapshotService(c.Snapshotter)
mounts, err := snapshotter.Mounts(ctx, c.SnapshotKey)
if err != nil {
return err
}
root, err := ioutil.TempDir("", "ctd-volume")
if err != nil {
return err
}
// We change RemoveAll to Remove so that we either leak a temp dir
// if it fails but not RM snapshot data.
// refer to https://github.com/containerd/containerd/pull/1868
// https://github.com/containerd/containerd/pull/1785
defer os.Remove(root) // nolint: errcheck
if err := mount.All(mounts, root); err != nil {
return errors.Wrap(err, "failed to mount")
}
defer func() {
if uerr := mount.Unmount(root, 0); uerr != nil {
log.G(ctx).WithError(uerr).Errorf("Failed to unmount snapshot %q", c.SnapshotKey)
if err == nil {
err = uerr
}
}
}()
for host, volume := range volumeMounts {
src := filepath.Join(root, volume)
if _, err := os.Stat(src); err != nil {
if os.IsNotExist(err) {
// Skip copying directory if it does not exist.
continue
}
return errors.Wrap(err, "stat volume in rootfs")
}
if err := copyExistingContents(src, host); err != nil {
return errors.Wrap(err, "taking runtime copy of volume")
}
}
return nil
}
}
// copyExistingContents copies from the source to the destination and
// ensures the ownership is appropriately set.
func copyExistingContents(source, destination string) error {
dstList, err := ioutil.ReadDir(destination)
if err != nil {
return err
}
if len(dstList) != 0 {
return errors.Errorf("volume at %q is not initially empty", destination)
}
return fs.CopyDir(destination, source)
}

113
pkg/containerd/opts/spec.go Normal file
View File

@ -0,0 +1,113 @@
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package opts
import (
"context"
"os"
"path/filepath"
"strings"
"github.com/containerd/containerd/containers"
"github.com/containerd/containerd/oci"
imagespec "github.com/opencontainers/image-spec/specs-go/v1"
runtimespec "github.com/opencontainers/runtime-spec/specs-go"
"github.com/pkg/errors"
runtime "k8s.io/cri-api/pkg/apis/runtime/v1alpha2"
)
// DefaultSandboxCPUshares is default cpu shares for sandbox container.
// TODO(windows): Revisit cpu shares for windows (https://github.com/containerd/cri/issues/1297)
const DefaultSandboxCPUshares = 2
// WithRelativeRoot sets the root for the container
func WithRelativeRoot(root string) oci.SpecOpts {
return func(ctx context.Context, client oci.Client, c *containers.Container, s *runtimespec.Spec) (err error) {
if s.Root == nil {
s.Root = &runtimespec.Root{}
}
s.Root.Path = root
return nil
}
}
// WithoutRoot sets the root to nil for the container.
func WithoutRoot(ctx context.Context, client oci.Client, c *containers.Container, s *runtimespec.Spec) error {
s.Root = nil
return nil
}
// WithProcessArgs sets the process args on the spec based on the image and runtime config
func WithProcessArgs(config *runtime.ContainerConfig, image *imagespec.ImageConfig) oci.SpecOpts {
return func(ctx context.Context, client oci.Client, c *containers.Container, s *runtimespec.Spec) (err error) {
command, args := config.GetCommand(), config.GetArgs()
// The following logic is migrated from https://github.com/moby/moby/blob/master/daemon/commit.go
// TODO(random-liu): Clearly define the commands overwrite behavior.
if len(command) == 0 {
// Copy array to avoid data race.
if len(args) == 0 {
args = append([]string{}, image.Cmd...)
}
if command == nil {
command = append([]string{}, image.Entrypoint...)
}
}
if len(command) == 0 && len(args) == 0 {
return errors.New("no command specified")
}
return oci.WithProcessArgs(append(command, args...)...)(ctx, client, c, s)
}
}
// mounts defines how to sort runtime.Mount.
// This is the same with the Docker implementation:
// https://github.com/moby/moby/blob/17.05.x/daemon/volumes.go#L26
type orderedMounts []*runtime.Mount
// Len returns the number of mounts. Used in sorting.
func (m orderedMounts) Len() int {
return len(m)
}
// Less returns true if the number of parts (a/b/c would be 3 parts) in the
// mount indexed by parameter 1 is less than that of the mount indexed by
// parameter 2. Used in sorting.
func (m orderedMounts) Less(i, j int) bool {
return m.parts(i) < m.parts(j)
}
// Swap swaps two items in an array of mounts. Used in sorting
func (m orderedMounts) Swap(i, j int) {
m[i], m[j] = m[j], m[i]
}
// parts returns the number of parts in the destination of a mount. Used in sorting.
func (m orderedMounts) parts(i int) int {
return strings.Count(filepath.Clean(m[i].ContainerPath), string(os.PathSeparator))
}
// WithAnnotation sets the provided annotation
func WithAnnotation(k, v string) oci.SpecOpts {
return func(ctx context.Context, client oci.Client, c *containers.Container, s *runtimespec.Spec) error {
if s.Annotations == nil {
s.Annotations = make(map[string]string)
}
s.Annotations[k] = v
return nil
}
}

719
pkg/containerd/opts/spec_linux.go Normal file
View File

@ -0,0 +1,719 @@
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package opts
import (
"context"
"fmt"
"io/ioutil"
"os"
"path/filepath"
"sort"
"strconv"
"strings"
"sync"
"syscall"
"github.com/containerd/containerd/containers"
"github.com/containerd/containerd/log"
"github.com/containerd/containerd/mount"
"github.com/containerd/containerd/oci"
"github.com/opencontainers/runc/libcontainer/devices"
runtimespec "github.com/opencontainers/runtime-spec/specs-go"
"github.com/opencontainers/selinux/go-selinux/label"
"github.com/pkg/errors"
"github.com/sirupsen/logrus"
"golang.org/x/sys/unix"
runtime "k8s.io/cri-api/pkg/apis/runtime/v1alpha2"
osinterface "github.com/containerd/cri/pkg/os"
"github.com/containerd/cri/pkg/util"
)
// WithAdditionalGIDs adds any additional groups listed for a particular user in the
// /etc/groups file of the image's root filesystem to the OCI spec's additionalGids array.
func WithAdditionalGIDs(userstr string) oci.SpecOpts {
return func(ctx context.Context, client oci.Client, c *containers.Container, s *runtimespec.Spec) (err error) {
if s.Process == nil {
s.Process = &runtimespec.Process{}
}
gids := s.Process.User.AdditionalGids
if err := oci.WithAdditionalGIDs(userstr)(ctx, client, c, s); err != nil {
return err
}
// Merge existing gids and new gids.
s.Process.User.AdditionalGids = mergeGids(s.Process.User.AdditionalGids, gids)
return nil
}
}
func mergeGids(gids1, gids2 []uint32) []uint32 {
gidsMap := make(map[uint32]struct{})
for _, gid1 := range gids1 {
gidsMap[gid1] = struct{}{}
}
for _, gid2 := range gids2 {
gidsMap[gid2] = struct{}{}
}
var gids []uint32
for gid := range gidsMap {
gids = append(gids, gid)
}
sort.Slice(gids, func(i, j int) bool { return gids[i] < gids[j] })
return gids
}
// WithoutRunMount removes the `/run` inside the spec
func WithoutRunMount(_ context.Context, _ oci.Client, c *containers.Container, s *runtimespec.Spec) error {
var (
mounts []runtimespec.Mount
current = s.Mounts
)
for _, m := range current {
if filepath.Clean(m.Destination) == "/run" {
continue
}
mounts = append(mounts, m)
}
s.Mounts = mounts
return nil
}
// WithoutDefaultSecuritySettings removes the default security settings generated on a spec
func WithoutDefaultSecuritySettings(_ context.Context, _ oci.Client, c *containers.Container, s *runtimespec.Spec) error {
if s.Process == nil {
s.Process = &runtimespec.Process{}
}
// Make sure no default seccomp/apparmor is specified
s.Process.ApparmorProfile = ""
if s.Linux != nil {
s.Linux.Seccomp = nil
}
// Remove default rlimits (See issue #515)
s.Process.Rlimits = nil
return nil
}
// WithMounts sorts and adds runtime and CRI mounts to the spec
func WithMounts(osi osinterface.OS, config *runtime.ContainerConfig, extra []*runtime.Mount, mountLabel string) oci.SpecOpts {
return func(ctx context.Context, client oci.Client, _ *containers.Container, s *runtimespec.Spec) (err error) {
// mergeMounts merge CRI mounts with extra mounts. If a mount destination
// is mounted by both a CRI mount and an extra mount, the CRI mount will
// be kept.
var (
criMounts = config.GetMounts()
mounts = append([]*runtime.Mount{}, criMounts...)
)
// Copy all mounts from extra mounts, except for mounts overridden by CRI.
for _, e := range extra {
found := false
for _, c := range criMounts {
if filepath.Clean(e.ContainerPath) == filepath.Clean(c.ContainerPath) {
found = true
break
}
}
if !found {
mounts = append(mounts, e)
}
}
// Sort mounts in number of parts. This ensures that high level mounts don't
// shadow other mounts.
sort.Sort(orderedMounts(mounts))
// Mount cgroup into the container as readonly, which inherits docker's behavior.
s.Mounts = append(s.Mounts, runtimespec.Mount{
Source: "cgroup",
Destination: "/sys/fs/cgroup",
Type: "cgroup",
Options: []string{"nosuid", "noexec", "nodev", "relatime", "ro"},
})
// Copy all mounts from default mounts, except for
// - mounts overridden by supplied mount;
// - all mounts under /dev if a supplied /dev is present.
mountSet := make(map[string]struct{})
for _, m := range mounts {
mountSet[filepath.Clean(m.ContainerPath)] = struct{}{}
}
defaultMounts := s.Mounts
s.Mounts = nil
for _, m := range defaultMounts {
dst := filepath.Clean(m.Destination)
if _, ok := mountSet[dst]; ok {
// filter out mount overridden by a supplied mount
continue
}
if _, mountDev := mountSet["/dev"]; mountDev && strings.HasPrefix(dst, "/dev/") {
// filter out everything under /dev if /dev is a supplied mount
continue
}
s.Mounts = append(s.Mounts, m)
}
for _, mount := range mounts {
var (
dst = mount.GetContainerPath()
src = mount.GetHostPath()
)
// Create the host path if it doesn't exist.
// TODO(random-liu): Add CRI validation test for this case.
if _, err := osi.Stat(src); err != nil {
if !os.IsNotExist(err) {
return errors.Wrapf(err, "failed to stat %q", src)
}
if err := osi.MkdirAll(src, 0755); err != nil {
return errors.Wrapf(err, "failed to mkdir %q", src)
}
}
// TODO(random-liu): Add cri-containerd integration test or cri validation test
// for this.
src, err := osi.ResolveSymbolicLink(src)
if err != nil {
return errors.Wrapf(err, "failed to resolve symlink %q", src)
}
if s.Linux == nil {
s.Linux = &runtimespec.Linux{}
}
options := []string{"rbind"}
switch mount.GetPropagation() {
case runtime.MountPropagation_PROPAGATION_PRIVATE:
options = append(options, "rprivate")
// Since default root propagation in runc is rprivate ignore
// setting the root propagation
case runtime.MountPropagation_PROPAGATION_BIDIRECTIONAL:
if err := ensureShared(src, osi.(osinterface.UNIX).LookupMount); err != nil {
return err
}
options = append(options, "rshared")
s.Linux.RootfsPropagation = "rshared"
case runtime.MountPropagation_PROPAGATION_HOST_TO_CONTAINER:
if err := ensureSharedOrSlave(src, osi.(osinterface.UNIX).LookupMount); err != nil {
return err
}
options = append(options, "rslave")
if s.Linux.RootfsPropagation != "rshared" &&
s.Linux.RootfsPropagation != "rslave" {
s.Linux.RootfsPropagation = "rslave"
}
default:
log.G(ctx).Warnf("Unknown propagation mode for hostPath %q", mount.HostPath)
options = append(options, "rprivate")
}
// NOTE(random-liu): we don't change all mounts to `ro` when root filesystem
// is readonly. This is different from docker's behavior, but make more sense.
if mount.GetReadonly() {
options = append(options, "ro")
} else {
options = append(options, "rw")
}
if mount.GetSelinuxRelabel() {
if err := label.Relabel(src, mountLabel, false); err != nil && err != unix.ENOTSUP {
return errors.Wrapf(err, "relabel %q with %q failed", src, mountLabel)
}
}
s.Mounts = append(s.Mounts, runtimespec.Mount{
Source: src,
Destination: dst,
Type: "bind",
Options: options,
})
}
return nil
}
}
// Ensure mount point on which path is mounted, is shared.
func ensureShared(path string, lookupMount func(string) (mount.Info, error)) error {
mountInfo, err := lookupMount(path)
if err != nil {
return err
}
// Make sure source mount point is shared.
optsSplit := strings.Split(mountInfo.Optional, " ")
for _, opt := range optsSplit {
if strings.HasPrefix(opt, "shared:") {
return nil
}
}
return errors.Errorf("path %q is mounted on %q but it is not a shared mount", path, mountInfo.Mountpoint)
}
// ensure mount point on which path is mounted, is either shared or slave.
func ensureSharedOrSlave(path string, lookupMount func(string) (mount.Info, error)) error {
mountInfo, err := lookupMount(path)
if err != nil {
return err
}
// Make sure source mount point is shared.
optsSplit := strings.Split(mountInfo.Optional, " ")
for _, opt := range optsSplit {
if strings.HasPrefix(opt, "shared:") {
return nil
} else if strings.HasPrefix(opt, "master:") {
return nil
}
}
return errors.Errorf("path %q is mounted on %q but it is not a shared or slave mount", path, mountInfo.Mountpoint)
}
func addDevice(s *runtimespec.Spec, rd runtimespec.LinuxDevice) {
for i, dev := range s.Linux.Devices {
if dev.Path == rd.Path {
s.Linux.Devices[i] = rd
return
}
}
s.Linux.Devices = append(s.Linux.Devices, rd)
}
// WithDevices sets the provided devices onto the container spec
func WithDevices(osi osinterface.OS, config *runtime.ContainerConfig) oci.SpecOpts {
return func(ctx context.Context, client oci.Client, c *containers.Container, s *runtimespec.Spec) (err error) {
if s.Linux == nil {
s.Linux = &runtimespec.Linux{}
}
if s.Linux.Resources == nil {
s.Linux.Resources = &runtimespec.LinuxResources{}
}
for _, device := range config.GetDevices() {
path, err := osi.ResolveSymbolicLink(device.HostPath)
if err != nil {
return err
}
dev, err := devices.DeviceFromPath(path, device.Permissions)
if err != nil {
return err
}
rd := runtimespec.LinuxDevice{
Path: device.ContainerPath,
Type: string(dev.Type),
Major: dev.Major,
Minor: dev.Minor,
UID: &dev.Uid,
GID: &dev.Gid,
}
addDevice(s, rd)
s.Linux.Resources.Devices = append(s.Linux.Resources.Devices, runtimespec.LinuxDeviceCgroup{
Allow: true,
Type: string(dev.Type),
Major: &dev.Major,
Minor: &dev.Minor,
Access: string(dev.Permissions),
})
}
return nil
}
}
// WithCapabilities sets the provided capabilties from the security context
func WithCapabilities(sc *runtime.LinuxContainerSecurityContext) oci.SpecOpts {
capabilities := sc.GetCapabilities()
if capabilities == nil {
return nullOpt
}
var opts []oci.SpecOpts
// Add/drop all capabilities if "all" is specified, so that
// following individual add/drop could still work. E.g.
// AddCapabilities: []string{"ALL"}, DropCapabilities: []string{"CHOWN"}
// will be all capabilities without `CAP_CHOWN`.
if util.InStringSlice(capabilities.GetAddCapabilities(), "ALL") {
opts = append(opts, oci.WithAllCapabilities)
}
if util.InStringSlice(capabilities.GetDropCapabilities(), "ALL") {
opts = append(opts, oci.WithCapabilities(nil))
}
var caps []string
for _, c := range capabilities.GetAddCapabilities() {
if strings.ToUpper(c) == "ALL" {
continue
}
// Capabilities in CRI doesn't have `CAP_` prefix, so add it.
caps = append(caps, "CAP_"+strings.ToUpper(c))
}
opts = append(opts, oci.WithAddedCapabilities(caps))
caps = []string{}
for _, c := range capabilities.GetDropCapabilities() {
if strings.ToUpper(c) == "ALL" {
continue
}
caps = append(caps, "CAP_"+strings.ToUpper(c))
}
opts = append(opts, oci.WithDroppedCapabilities(caps))
return oci.Compose(opts...)
}
// WithoutAmbientCaps removes the ambient caps from the spec
func WithoutAmbientCaps(_ context.Context, _ oci.Client, c *containers.Container, s *runtimespec.Spec) error {
if s.Process == nil {
s.Process = &runtimespec.Process{}
}
if s.Process.Capabilities == nil {
s.Process.Capabilities = &runtimespec.LinuxCapabilities{}
}
s.Process.Capabilities.Ambient = nil
return nil
}
// WithDisabledCgroups clears the Cgroups Path from the spec
func WithDisabledCgroups(_ context.Context, _ oci.Client, c *containers.Container, s *runtimespec.Spec) error {
if s.Linux == nil {
s.Linux = &runtimespec.Linux{}
}
s.Linux.CgroupsPath = ""
return nil
}
// WithSelinuxLabels sets the mount and process labels
func WithSelinuxLabels(process, mount string) oci.SpecOpts {
return func(ctx context.Context, client oci.Client, c *containers.Container, s *runtimespec.Spec) (err error) {
if s.Linux == nil {
s.Linux = &runtimespec.Linux{}
}
if s.Process == nil {
s.Process = &runtimespec.Process{}
}
s.Linux.MountLabel = mount
s.Process.SelinuxLabel = process
return nil
}
}
// WithResources sets the provided resource restrictions
func WithResources(resources *runtime.LinuxContainerResources, tolerateMissingHugetlbController, disableHugetlbController bool) oci.SpecOpts {
return func(ctx context.Context, client oci.Client, c *containers.Container, s *runtimespec.Spec) (err error) {
if resources == nil {
return nil
}
if s.Linux == nil {
s.Linux = &runtimespec.Linux{}
}
if s.Linux.Resources == nil {
s.Linux.Resources = &runtimespec.LinuxResources{}
}
if s.Linux.Resources.CPU == nil {
s.Linux.Resources.CPU = &runtimespec.LinuxCPU{}
}
if s.Linux.Resources.Memory == nil {
s.Linux.Resources.Memory = &runtimespec.LinuxMemory{}
}
var (
p = uint64(resources.GetCpuPeriod())
q = resources.GetCpuQuota()
shares = uint64(resources.GetCpuShares())
limit = resources.GetMemoryLimitInBytes()
hugepages = resources.GetHugepageLimits()
)
if p != 0 {
s.Linux.Resources.CPU.Period = &p
}
if q != 0 {
s.Linux.Resources.CPU.Quota = &q
}
if shares != 0 {
s.Linux.Resources.CPU.Shares = &shares
}
if cpus := resources.GetCpusetCpus(); cpus != "" {
s.Linux.Resources.CPU.Cpus = cpus
}
if mems := resources.GetCpusetMems(); mems != "" {
s.Linux.Resources.CPU.Mems = resources.GetCpusetMems()
}
if limit != 0 {
s.Linux.Resources.Memory.Limit = &limit
}
if !disableHugetlbController {
if isHugetlbControllerPresent() {
for _, limit := range hugepages {
s.Linux.Resources.HugepageLimits = append(s.Linux.Resources.HugepageLimits, runtimespec.LinuxHugepageLimit{
Pagesize: limit.PageSize,
Limit: limit.Limit,
})
}
} else {
if !tolerateMissingHugetlbController {
return errors.Errorf("huge pages limits are specified but hugetlb cgroup controller is missing. " +
"Please set tolerate_missing_hugetlb_controller to `true` to ignore this error")
}
logrus.Warn("hugetlb cgroup controller is absent. skipping huge pages limits")
}
}
return nil
}
}
var (
supportsHugetlbOnce sync.Once
supportsHugetlb bool
)
func isHugetlbControllerPresent() bool {
supportsHugetlbOnce.Do(func() {
supportsHugetlb = false
if IsCgroup2UnifiedMode() {
supportsHugetlb, _ = cgroupv2HasHugetlb()
} else {
supportsHugetlb, _ = cgroupv1HasHugetlb()
}
})
return supportsHugetlb
}
var (
_cgroupv1HasHugetlbOnce sync.Once
_cgroupv1HasHugetlb bool
_cgroupv1HasHugetlbErr error
_cgroupv2HasHugetlbOnce sync.Once
_cgroupv2HasHugetlb bool
_cgroupv2HasHugetlbErr error
isUnifiedOnce sync.Once
isUnified bool
)
// cgroupv1HasHugetlb returns whether the hugetlb controller is present on
// cgroup v1.
func cgroupv1HasHugetlb() (bool, error) {
_cgroupv1HasHugetlbOnce.Do(func() {
if _, err := ioutil.ReadDir("/sys/fs/cgroup/hugetlb"); err != nil {
_cgroupv1HasHugetlbErr = errors.Wrap(err, "readdir /sys/fs/cgroup/hugetlb")
_cgroupv1HasHugetlb = false
} else {
_cgroupv1HasHugetlbErr = nil
_cgroupv1HasHugetlb = true
}
})
return _cgroupv1HasHugetlb, _cgroupv1HasHugetlbErr
}
// cgroupv2HasHugetlb returns whether the hugetlb controller is present on
// cgroup v2.
func cgroupv2HasHugetlb() (bool, error) {
_cgroupv2HasHugetlbOnce.Do(func() {
controllers, err := ioutil.ReadFile("/sys/fs/cgroup/cgroup.controllers")
if err != nil {
_cgroupv2HasHugetlbErr = errors.Wrap(err, "read /sys/fs/cgroup/cgroup.controllers")
return
}
_cgroupv2HasHugetlb = strings.Contains(string(controllers), "hugetlb")
})
return _cgroupv2HasHugetlb, _cgroupv2HasHugetlbErr
}
// IsCgroup2UnifiedMode returns whether we are running in cgroup v2 unified mode.
func IsCgroup2UnifiedMode() bool {
isUnifiedOnce.Do(func() {
var st syscall.Statfs_t
if err := syscall.Statfs("/sys/fs/cgroup", &st); err != nil {
panic("cannot statfs cgroup root")
}
isUnified = st.Type == unix.CGROUP2_SUPER_MAGIC
})
return isUnified
}
// WithOOMScoreAdj sets the oom score
func WithOOMScoreAdj(config *runtime.ContainerConfig, restrict bool) oci.SpecOpts {
return func(ctx context.Context, client oci.Client, c *containers.Container, s *runtimespec.Spec) error {
if s.Process == nil {
s.Process = &runtimespec.Process{}
}
resources := config.GetLinux().GetResources()
if resources == nil {
return nil
}
adj := int(resources.GetOomScoreAdj())
if restrict {
var err error
adj, err = restrictOOMScoreAdj(adj)
if err != nil {
return err
}
}
s.Process.OOMScoreAdj = &adj
return nil
}
}
// WithSysctls sets the provided sysctls onto the spec
func WithSysctls(sysctls map[string]string) oci.SpecOpts {
return func(ctx context.Context, client oci.Client, c *containers.Container, s *runtimespec.Spec) error {
if s.Linux == nil {
s.Linux = &runtimespec.Linux{}
}
if s.Linux.Sysctl == nil {
s.Linux.Sysctl = make(map[string]string)
}
for k, v := range sysctls {
s.Linux.Sysctl[k] = v
}
return nil
}
}
// WithPodOOMScoreAdj sets the oom score for the pod sandbox
func WithPodOOMScoreAdj(adj int, restrict bool) oci.SpecOpts {
return func(ctx context.Context, client oci.Client, c *containers.Container, s *runtimespec.Spec) error {
if s.Process == nil {
s.Process = &runtimespec.Process{}
}
if restrict {
var err error
adj, err = restrictOOMScoreAdj(adj)
if err != nil {
return err
}
}
s.Process.OOMScoreAdj = &adj
return nil
}
}
// WithSupplementalGroups sets the supplemental groups for the process
func WithSupplementalGroups(groups []int64) oci.SpecOpts {
return func(ctx context.Context, client oci.Client, c *containers.Container, s *runtimespec.Spec) error {
if s.Process == nil {
s.Process = &runtimespec.Process{}
}
var guids []uint32
for _, g := range groups {
guids = append(guids, uint32(g))
}
s.Process.User.AdditionalGids = mergeGids(s.Process.User.AdditionalGids, guids)
return nil
}
}
// WithPodNamespaces sets the pod namespaces for the container
func WithPodNamespaces(config *runtime.LinuxContainerSecurityContext, pid uint32) oci.SpecOpts {
namespaces := config.GetNamespaceOptions()
opts := []oci.SpecOpts{
oci.WithLinuxNamespace(runtimespec.LinuxNamespace{Type: runtimespec.NetworkNamespace, Path: GetNetworkNamespace(pid)}),
oci.WithLinuxNamespace(runtimespec.LinuxNamespace{Type: runtimespec.IPCNamespace, Path: GetIPCNamespace(pid)}),
oci.WithLinuxNamespace(runtimespec.LinuxNamespace{Type: runtimespec.UTSNamespace, Path: GetUTSNamespace(pid)}),
}
if namespaces.GetPid() != runtime.NamespaceMode_CONTAINER {
opts = append(opts, oci.WithLinuxNamespace(runtimespec.LinuxNamespace{Type: runtimespec.PIDNamespace, Path: GetPIDNamespace(pid)}))
}
return oci.Compose(opts...)
}
// WithDefaultSandboxShares sets the default sandbox CPU shares
func WithDefaultSandboxShares(ctx context.Context, client oci.Client, c *containers.Container, s *runtimespec.Spec) error {
if s.Linux == nil {
s.Linux = &runtimespec.Linux{}
}
if s.Linux.Resources == nil {
s.Linux.Resources = &runtimespec.LinuxResources{}
}
if s.Linux.Resources.CPU == nil {
s.Linux.Resources.CPU = &runtimespec.LinuxCPU{}
}
i := uint64(DefaultSandboxCPUshares)
s.Linux.Resources.CPU.Shares = &i
return nil
}
// WithoutNamespace removes the provided namespace
func WithoutNamespace(t runtimespec.LinuxNamespaceType) oci.SpecOpts {
return func(ctx context.Context, client oci.Client, c *containers.Container, s *runtimespec.Spec) error {
if s.Linux == nil {
return nil
}
var namespaces []runtimespec.LinuxNamespace
for i, ns := range s.Linux.Namespaces {
if ns.Type != t {
namespaces = append(namespaces, s.Linux.Namespaces[i])
}
}
s.Linux.Namespaces = namespaces
return nil
}
}
func nullOpt(_ context.Context, _ oci.Client, _ *containers.Container, _ *runtimespec.Spec) error {
return nil
}
func getCurrentOOMScoreAdj() (int, error) {
b, err := ioutil.ReadFile("/proc/self/oom_score_adj")
if err != nil {
return 0, errors.Wrap(err, "could not get the daemon oom_score_adj")
}
s := strings.TrimSpace(string(b))
i, err := strconv.Atoi(s)
if err != nil {
return 0, errors.Wrap(err, "could not get the daemon oom_score_adj")
}
return i, nil
}
func restrictOOMScoreAdj(preferredOOMScoreAdj int) (int, error) {
currentOOMScoreAdj, err := getCurrentOOMScoreAdj()
if err != nil {
return preferredOOMScoreAdj, err
}
if preferredOOMScoreAdj < currentOOMScoreAdj {
return currentOOMScoreAdj, nil
}
return preferredOOMScoreAdj, nil
}
const (
// netNSFormat is the format of network namespace of a process.
netNSFormat = "/proc/%v/ns/net"
// ipcNSFormat is the format of ipc namespace of a process.
ipcNSFormat = "/proc/%v/ns/ipc"
// utsNSFormat is the format of uts namespace of a process.
utsNSFormat = "/proc/%v/ns/uts"
// pidNSFormat is the format of pid namespace of a process.
pidNSFormat = "/proc/%v/ns/pid"
)
// GetNetworkNamespace returns the network namespace of a process.
func GetNetworkNamespace(pid uint32) string {
return fmt.Sprintf(netNSFormat, pid)
}
// GetIPCNamespace returns the ipc namespace of a process.
func GetIPCNamespace(pid uint32) string {
return fmt.Sprintf(ipcNSFormat, pid)
}
// GetUTSNamespace returns the uts namespace of a process.
func GetUTSNamespace(pid uint32) string {
return fmt.Sprintf(utsNSFormat, pid)
}
// GetPIDNamespace returns the pid namespace of a process.
func GetPIDNamespace(pid uint32) string {
return fmt.Sprintf(pidNSFormat, pid)
}

47
pkg/containerd/opts/spec_linux_test.go Normal file
View File

@ -0,0 +1,47 @@
/*
Copyright The containerd Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package opts
import (
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestMergeGids(t *testing.T) {
gids1 := []uint32{3, 2, 1}
gids2 := []uint32{2, 3, 4}
assert.Equal(t, []uint32{1, 2, 3, 4}, mergeGids(gids1, gids2))
}
func TestRestrictOOMScoreAdj(t *testing.T) {
current, err := getCurrentOOMScoreAdj()
require.NoError(t, err)
got, err := restrictOOMScoreAdj(current - 1)
require.NoError(t, err)
assert.Equal(t, got, current)
got, err = restrictOOMScoreAdj(current)
require.NoError(t, err)
assert.Equal(t, got, current)
got, err = restrictOOMScoreAdj(current + 1)
require.NoError(t, err)
assert.Equal(t, got, current+1)
}

Some files were not shown because too many files have changed in this diff Show More