Go to file
Michael Crosby e6ae9cc64f Shim pluggable logging
Closes #603

This adds logging facilities at the shim level to provide minimal I/O
overhead and pluggable logging options.  Log handling is done within the
shim so that all I/O, cpu, and memory can be charged to the container.

A sample logging driver setting up logging for a container the systemd
journal looks like this:

```go
package main

import (
	"bufio"
	"context"
	"fmt"
	"io"
	"sync"

	"github.com/containerd/containerd/runtime/v2/logging"
	"github.com/coreos/go-systemd/journal"
)

func main() {
	logging.Run(log)
}

func log(ctx context.Context, config *logging.Config, ready func() error) error {
	// construct any log metadata for the container
	vars := map[string]string{
		"SYSLOG_IDENTIFIER": fmt.Sprintf("%s:%s", config.Namespace, config.ID),
	}
	var wg sync.WaitGroup
	wg.Add(2)
	// forward both stdout and stderr to the journal
	go copy(&wg, config.Stdout, journal.PriInfo, vars)
	go copy(&wg, config.Stderr, journal.PriErr, vars)

	// signal that we are ready and setup for the container to be started
	if err := ready(); err != nil {
		return err
	}
	wg.Wait()
	return nil
}

func copy(wg *sync.WaitGroup, r io.Reader, pri journal.Priority, vars map[string]string) {
	defer wg.Done()
	s := bufio.NewScanner(r)
	for s.Scan() {
		if s.Err() != nil {
			return
		}
		journal.Send(s.Text(), pri, vars)
	}
}
```

A `logging` package has been created to assist log developers create
logging plugins for containerd.

This uses a URI based approach for logging drivers that can be expanded
in the future.

Supported URI scheme's are:

* binary
* fifo
* file

You can pass the log url via ctr on the command line:

```bash
> ctr run --rm --runtime io.containerd.runc.v2 --log-uri binary://shim-journald docker.io/library/redis:alpine redis
```

```bash
> journalctl -f -t default:redis

-- Logs begin at Tue 2018-12-11 16:29:51 EST. --
Mar 08 16:08:22 deathstar default:redis[120760]: 1:C 08 Mar 2019 21:08:22.703 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
Mar 08 16:08:22 deathstar default:redis[120760]: 1:M 08 Mar 2019 21:08:22.704 # You requested maxclients of 10000 requiring at least 10032 max file descriptors.
Mar 08 16:08:22 deathstar default:redis[120760]: 1:M 08 Mar 2019 21:08:22.704 # Server can't set maximum open files to 10032 because of OS error: Operation not permitted.
Mar 08 16:08:22 deathstar default:redis[120760]: 1:M 08 Mar 2019 21:08:22.704 # Current maximum open files is 1024. maxclients has been reduced to 992 to compensate for low ulimit. If you need higher maxclients increase 'ulimit -n'.
Mar 08 16:08:22 deathstar default:redis[120760]: 1:M 08 Mar 2019 21:08:22.705 * Running mode=standalone, port=6379.
Mar 08 16:08:22 deathstar default:redis[120760]: 1:M 08 Mar 2019 21:08:22.705 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
Mar 08 16:08:22 deathstar default:redis[120760]: 1:M 08 Mar 2019 21:08:22.705 # Server initialized
Mar 08 16:08:22 deathstar default:redis[120760]: 1:M 08 Mar 2019 21:08:22.705 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
Mar 08 16:08:22 deathstar default:redis[120760]: 1:M 08 Mar 2019 21:08:22.705 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
Mar 08 16:08:22 deathstar default:redis[120760]: 1:M 08 Mar 2019 21:08:22.705 * Ready to accept connections
Mar 08 16:08:50 deathstar default:redis[120760]: 1:signal-handler (1552079330) Received SIGINT scheduling shutdown...
Mar 08 16:08:50 deathstar default:redis[120760]: 1:M 08 Mar 2019 21:08:50.405 # User requested shutdown...
Mar 08 16:08:50 deathstar default:redis[120760]: 1:M 08 Mar 2019 21:08:50.406 * Saving the final RDB snapshot before exiting.
Mar 08 16:08:50 deathstar default:redis[120760]: 1:M 08 Mar 2019 21:08:50.452 * DB saved on disk
Mar 08 16:08:50 deathstar default:redis[120760]: 1:M 08 Mar 2019 21:08:50.453 # Redis is now ready to exit, bye bye...
```

The following client side Opts are added:

```go
// LogURI provides the raw logging URI
func LogURI(uri *url.URL) Creator { }
// BinaryIO forwards contianer STDOUT|STDERR directly to a logging binary
func BinaryIO(binary string, args map[string]string) Creator {}
```

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2019-03-12 12:18:28 -04:00
.github Add github issue template 2018-01-17 11:11:53 -05:00
api Add support for TaskDelete event on exec in RuntimeV2 2019-02-12 15:50:21 -08:00
archive Unpack should set 0755 when the parent directory doesn't exist. 2019-02-14 13:38:12 -08:00
cio Shim pluggable logging 2019-03-12 12:18:28 -04:00
cmd Shim pluggable logging 2019-03-12 12:18:28 -04:00
containers fixing typo and added a missing comment. 2019-02-18 14:13:02 -08:00
content Merge pull request #3008 from JoeWrightss/patch-3 2019-02-13 09:58:37 -08:00
contrib fix: fix error info start capitalized 2018-11-28 15:26:16 +08:00
defaults fix words misspell 2018-04-23 00:09:42 -04:00
design Remove outdated plugin design doc 2018-07-25 23:28:09 -07:00
diff fix: linter issue 2019-01-23 22:54:51 +08:00
docs docs: Add NAME section in all manpages 2019-02-22 23:40:28 +08:00
errdefs Eliminate misuses of errors.Wrapf 2018-11-29 14:26:53 -08:00
events refactor: no need defer in closeAll 2018-11-19 09:41:38 -05:00
filters fix: linter issue 2019-01-23 22:54:51 +08:00
gc Update tests to use gotest.tools angel 2018-06-08 21:02:01 +02:00
identifiers Fix annotation typo errors 2018-12-31 22:40:06 +08:00
images Use distribution's reference.ParseDockerRef 2019-02-07 13:13:25 +01:00
labels Licence header added 2018-02-19 10:32:26 +09:00
leases Update lease service errors 2018-07-25 14:56:35 -07:00
log Set default log formatting to use RFC3339Nano with fixed width 2018-07-24 10:23:38 +02:00
metadata Add content gc ref labels from containers, images, and snapshots 2019-03-04 14:51:07 -08:00
metrics/cgroups allow idempotence when adding a task to cgroup metrics collection 2018-10-18 01:32:56 -05:00
mount fix: fix error info start capitalized 2018-11-28 15:26:16 +08:00
namespaces Switch from x/net/context to context 2018-04-24 14:33:34 -07:00
oci Add runc.v2 multi-shim 2019-02-21 11:09:46 -05:00
pkg remove excessive []byte(s) conversion 2019-01-28 19:50:28 +03:00
platforms Add support to detect ARM variant on Windows 2018-10-10 14:17:11 -07:00
plugin Add runc.v2 multi-shim 2019-02-21 11:09:46 -05:00
protobuf/plugin protobuf: remove generated google/rpc files 2018-04-05 14:36:32 -07:00
reference Licence header added 2018-02-19 10:32:26 +09:00
releases Add website update to release process 2019-02-14 14:01:00 -05:00
remotes Add image handler wrapper 2019-02-12 16:34:06 -08:00
reports fix typo 2018-09-07 15:35:57 +08:00
rootfs Switching from crypto/rand to math/rand to avoid blocking 2018-07-11 16:24:51 -04:00
runtime Shim pluggable logging 2019-03-12 12:18:28 -04:00
script/setup Update cri-tools to v1.13.0. 2019-01-02 12:54:55 -08:00
services Add runc.v2 multi-shim 2019-02-21 11:09:46 -05:00
snapshots Update the snapshotter docs to refer to the snapshots package for 2019-02-28 15:03:37 -08:00
sys fix: fix error info start capitalized 2018-11-28 15:26:16 +08:00
vendor Merge pull request #3041 from thaJeztah/bump_da_bolt 2019-03-07 09:28:26 -08:00
version Prepare 1.2 release 2018-10-24 16:14:46 -07:00
windows runtime: add Add/Delete method in PlatformRuntime interface 2018-12-29 13:56:38 +08:00
.appveyor.yml Fix mingw version back to working version with Golang 2018-11-02 09:15:51 -04:00
.gitignore Add capability to build manpages from markdown 2018-01-30 19:50:45 -05:00
.gometalinter.json fix: linter issue 2019-01-23 22:54:51 +08:00
.mailmap Update mailmap 2018-09-20 11:23:42 -07:00
.travis.yml Merge pull request #3040 from crosbymichael/travis-bump 2019-02-21 16:21:37 -08:00
ADOPTERS.md Add more adopters 2018-11-29 04:53:52 -05:00
benchmark_test.go Tests should set up snapshot prior to any use of fs 2018-09-14 14:36:01 -04:00
BUILDING.md BUILDING.md: update testing section 2019-03-06 18:56:14 +08:00
client_opts.go Add image handler wrapper 2019-02-12 16:34:06 -08:00
client_test.go Add runc.v2 multi-shim 2019-02-21 11:09:46 -05:00
client_unix_test.go Licence header added 2018-02-19 10:32:26 +09:00
client_windows_test.go Licence header added 2018-02-19 10:32:26 +09:00
client.go Add runc.v2 multi-shim 2019-02-21 11:09:46 -05:00
code-of-conduct.md Add code of conduct 2017-04-04 09:41:38 -07:00
container_checkpoint_opts.go checkpoint: add copts to checkpoint; save snapshotter to annotation 2018-11-12 11:48:16 +00:00
container_checkpoint_test.go Update checkpoint opts with runtime handling 2019-03-06 12:42:45 -05:00
container_linux_test.go Add runc.v2 multi-shim 2019-02-21 11:09:46 -05:00
container_opts_unix.go update tests 2018-11-12 11:47:17 +00:00
container_opts.go Allow WithNewSnapshot and WithNewSnapshotView to take in snapshotter options. 2019-03-06 13:22:37 -08:00
container_restore_opts.go checkpoint: add copts to checkpoint; save snapshotter to annotation 2018-11-12 11:48:16 +00:00
container_test.go fix: linter issue 2019-01-23 22:54:51 +08:00
container.go Fix no pivot and keyring opts 2019-03-06 12:37:36 -05:00
containerd.service Ignore modprobe failures in ExecStartPre (systemd unit) 2018-11-10 12:52:06 +01:00
containerstore.go Add containers streaming API 2018-07-26 13:55:20 -04:00
content_test.go Pass in context to lease done function in client 2018-03-22 14:09:24 -07:00
daemon_config_linux_test.go Add runc.v2 multi-shim 2019-02-21 11:09:46 -05:00
daemon_test.go linux: fix runtime-root propagation 2018-03-05 15:01:22 +09:00
diff.go Add service plugin and support in process integration. 2018-03-12 18:03:50 +00:00
events.go bugfix: return the context error during event subscribe 2018-10-24 19:25:57 +08:00
export_test.go fix ctr image export not found error 2018-12-17 08:52:28 +08:00
export.go fix ctr image export not found error 2018-12-17 08:52:28 +08:00
grpc.go Switch from x/net/context to context 2018-04-24 14:33:34 -07:00
helpers_unix_test.go Licence header added 2018-02-19 10:32:26 +09:00
helpers_windows_test.go Licence header added 2018-02-19 10:32:26 +09:00
image_store.go Licence header added 2018-02-19 10:32:26 +09:00
image_test.go Update multi-arch image tests 2018-08-01 11:22:07 -07:00
image.go bugfix: unpack should always set the snapshot gc label 2018-12-14 09:59:19 +08:00
import_test.go fix ctr image export not found error 2018-12-17 08:52:28 +08:00
import.go fix: linter issue 2019-01-23 22:54:51 +08:00
install_opts.go Add optional install path 2018-09-11 10:36:00 -04:00
install.go fix: fix defer in loop 2018-12-16 13:16:23 +08:00
lease.go *: replace 3600 seconds with 1 hour 2018-08-02 11:40:03 -07:00
LICENSE Edit Containerd license info so GitHub recognizes it 2018-04-02 12:31:15 -07:00
Makefile Add runc.v2 multi-shim 2019-02-21 11:09:46 -05:00
Makefile.darwin Licence header added 2018-02-19 10:32:26 +09:00
Makefile.freebsd Licence header added 2018-02-19 10:32:26 +09:00
Makefile.linux Add runc.v2 multi-shim 2019-02-21 11:09:46 -05:00
Makefile.windows Build Windows V1 and V2 runtimes always 2018-11-02 11:59:03 -07:00
namespaces.go Licence header added 2018-02-19 10:32:26 +09:00
NOTICE Update readme and documentation for release 2015-12-16 12:15:22 -08:00
PLUGINS.md Document plugins 2018-07-25 23:28:09 -07:00
process.go Fix potential containerd panic. 2019-02-05 00:38:02 -08:00
Protobuild.toml Add io.containerd.runhcs.v1 shim proto options 2018-11-05 09:08:48 -08:00
pull.go Add image handler wrapper 2019-02-12 16:34:06 -08:00
README.md readme: fix example for checkpoint 2019-02-02 19:57:36 +08:00
RELEASES.md Update supported versions 2019-02-14 10:54:16 -08:00
ROADMAP.md Update roadmap.md with issue labels and milestones 2018-01-04 10:47:06 -05:00
RUNC.md fix RUNC.md vs vendor.conf mismatch 2018-05-11 13:56:54 +09:00
SCOPE.md Update readme 2017-06-20 13:19:53 -07:00
services.go Add leases manager interface 2018-07-18 10:43:37 -07:00
signals_test.go Use unix.SignalNum in ParseSignal on unix platform 2019-03-03 22:10:58 +08:00
signals_unix.go Use unix.SignalNum in ParseSignal on unix platform 2019-03-03 22:10:58 +08:00
signals_windows.go Use unix.SignalNum in ParseSignal on unix platform 2019-03-03 22:10:58 +08:00
signals.go Use unix.SignalNum in ParseSignal on unix platform 2019-03-03 22:10:58 +08:00
snapshot_test.go Licence header added 2018-02-19 10:32:26 +09:00
snapshotter_default_linux.go Licence header added 2018-02-19 10:32:26 +09:00
snapshotter_default_unix.go enable native (formerly naive) snapshotter by default 2018-04-02 13:29:19 +09:00
snapshotter_default_windows.go Licence header added 2018-02-19 10:32:26 +09:00
task_opts_unix_test.go move WithXXXX to task_opts.go 2018-09-10 15:24:01 +08:00
task_opts_unix.go Fix no pivot and keyring opts 2019-03-06 12:37:36 -05:00
task_opts.go Update checkpoint opts with runtime handling 2019-03-06 12:42:45 -05:00
task.go Update checkpoint opts with runtime handling 2019-03-06 12:42:45 -05:00
vendor.conf Merge pull request #3041 from thaJeztah/bump_da_bolt 2019-03-07 09:28:26 -08:00

containerd banner

GoDoc Build Status Windows Build Status FOSSA Status Go Report Card CII Best Practices

containerd is an industry-standard container runtime with an emphasis on simplicity, robustness and portability. It is available as a daemon for Linux and Windows, which can manage the complete container lifecycle of its host system: image transfer and storage, container execution and supervision, low-level storage and network attachments, etc.

containerd is designed to be embedded into a larger system, rather than being used directly by developers or end-users.

architecture

Getting Started

See our documentation on containerd.io:

See how to build containerd from source at BUILDING.

If you are interested in trying out containerd see our example at Getting Started.

Runtime Requirements

Runtime requirements for containerd are very minimal. Most interactions with the Linux and Windows container feature sets are handled via runc and/or OS-specific libraries (e.g. hcsshim for Microsoft). The current required version of runc is always listed in RUNC.md.

There are specific features used by containerd core code and snapshotters that will require a minimum kernel version on Linux. With the understood caveat of distro kernel versioning, a reasonable starting point for Linux is a minimum 4.x kernel version.

The overlay filesystem snapshotter, used by default, uses features that were finalized in the 4.x kernel series. If you choose to use btrfs, there may be more flexibility in kernel version (minimum recommended is 3.18), but will require the btrfs kernel module and btrfs tools to be installed on your Linux distribution.

To use Linux checkpoint and restore features, you will need criu installed on your system. See more details in Checkpoint and Restore.

Build requirements for developers are listed in BUILDING.

Features

Client

containerd offers a full client package to help you integrate containerd into your platform.


import (
  "github.com/containerd/containerd"
  "github.com/containerd/containerd/cio"
)


func main() {
	client, err := containerd.New("/run/containerd/containerd.sock")
	defer client.Close()
}

Namespaces

Namespaces allow multiple consumers to use the same containerd without conflicting with each other. It has the benefit of sharing content but still having separation with containers and images.

To set a namespace for requests to the API:

context = context.Background()
// create a context for docker
docker = namespaces.WithNamespace(context, "docker")

containerd, err := client.NewContainer(docker, "id")

To set a default namespace on the client:

client, err := containerd.New(address, containerd.WithDefaultNamespace("docker"))

Distribution

// pull an image
image, err := client.Pull(context, "docker.io/library/redis:latest")

// push an image
err := client.Push(context, "docker.io/library/redis:latest", image.Target())

Containers

In containerd, a container is a metadata object. Resources such as an OCI runtime specification, image, root filesystem, and other metadata can be attached to a container.

redis, err := client.NewContainer(context, "redis-master")
defer redis.Delete(context)

OCI Runtime Specification

containerd fully supports the OCI runtime specification for running containers. We have built in functions to help you generate runtime specifications based on images as well as custom parameters.

You can specify options when creating a container about how to modify the specification.

redis, err := client.NewContainer(context, "redis-master", containerd.WithNewSpec(oci.WithImageConfig(image)))

Root Filesystems

containerd allows you to use overlay or snapshot filesystems with your containers. It comes with builtin support for overlayfs and btrfs.

// pull an image and unpack it into the configured snapshotter
image, err := client.Pull(context, "docker.io/library/redis:latest", containerd.WithPullUnpack)

// allocate a new RW root filesystem for a container based on the image
redis, err := client.NewContainer(context, "redis-master",
	containerd.WithNewSnapshot("redis-rootfs", image),
	containerd.WithNewSpec(oci.WithImageConfig(image)),
)

// use a readonly filesystem with multiple containers
for i := 0; i < 10; i++ {
	id := fmt.Sprintf("id-%s", i)
	container, err := client.NewContainer(ctx, id,
		containerd.WithNewSnapshotView(id, image),
		containerd.WithNewSpec(oci.WithImageConfig(image)),
	)
}

Tasks

Taking a container object and turning it into a runnable process on a system is done by creating a new Task from the container. A task represents the runnable object within containerd.

// create a new task
task, err := redis.NewTask(context, cio.Stdio)
defer task.Delete(context)

// the task is now running and has a pid that can be use to setup networking
// or other runtime settings outside of containerd
pid := task.Pid()

// start the redis-server process inside the container
err := task.Start(context)

// wait for the task to exit and get the exit status
status, err := task.Wait(context)

Checkpoint and Restore

If you have criu installed on your machine you can checkpoint and restore containers and their tasks. This allow you to clone and/or live migrate containers to other machines.

// checkpoint the task then push it to a registry
checkpoint, err := task.Checkpoint(context)

err := client.Push(context, "myregistry/checkpoints/redis:master", checkpoint)

// on a new machine pull the checkpoint and restore the redis container
checkpoint, err := client.Pull(context, "myregistry/checkpoints/redis:master")

redis, err = client.NewContainer(context, "redis-master", containerd.WithNewSnapshot("redis-rootfs", checkpoint))
defer container.Delete(context)

task, err = redis.NewTask(context, cio.Stdio, containerd.WithTaskCheckpoint(checkpoint))
defer task.Delete(context)

err := task.Start(context)

Snapshot Plugins

In addition to the built-in Snapshot plugins in containerd, additional external plugins can be configured using GRPC. An external plugin is made available using the configured name and appears as a plugin alongside the built-in ones.

To add an external snapshot plugin, add the plugin to containerd's config file (by default at /etc/containerd/config.toml). The string following proxy_plugin. will be used as the name of the snapshotter and the address should refer to a socket with a GRPC listener serving containerd's Snapshot GRPC API. Remember to restart containerd for any configuration changes to take effect.

[proxy_plugins]
  [proxy_plugins.customsnapshot]
    type = "snapshot"
    address =  "/var/run/mysnapshotter.sock"

See PLUGINS.md for how to create plugins

Releases and API Stability

Please see RELEASES.md for details on versioning and stability of containerd components.

Communication

For async communication and long running discussions please use issues and pull requests on the github repo. This will be the best place to discuss design and implementation.

For sync communication we have a community slack with a #containerd channel that everyone is welcome to join and chat about development.

Slack: Catch us in the #containerd and #containerd-dev channels on dockercommunity.slack.com. Click here for an invite to docker community slack.

Security audit

A third party security audit was performed by Cure53 in 4Q2018; the full report is available in our docs/ directory.

Reporting security issues

If you are reporting a security issue, please reach out discreetly at security@containerd.io.

Licenses

The containerd codebase is released under the Apache 2.0 license. The README.md file, and files in the "docs" folder are licensed under the Creative Commons Attribution 4.0 International License. You may obtain a copy of the license, titled CC-BY-4.0, at http://creativecommons.org/licenses/by/4.0/.

Project details

containerd is the primary open source project within the broader containerd GitHub repository. However, all projects within the repo have common maintainership, governance, and contributing guidelines which are stored in a project repository commonly for all containerd projects.

Please find all these core project documents, including the:

information in our containerd/project repository.

Adoption

Interested to see who is using containerd? Are you using containerd in a project? Please add yourself via pull request to our ADOPTERS.md file.