btrfs plugin needs CGO support. However on riscv64, cgo
is only support on go1.16 (not released yet).
Instead of setting no_btrfs manually, adding a cgo tag tells
the compiler to skip it automatically.
Signed-off-by: Shengjing Zhu <zhsj@debian.org>
Go example:
```go
opts := []converter.Opt{
// convert Docker media types to OCI ones
converter.WithDocker2OCI(true),
// convert tar.gz layers to uncompressed tar layers
converter.WithLayerConvertFunc(uncompress.LayerConvertFunc),
}
srcRef := "example.com/foo:orig"
dstRef := "example.com/foo:converted"
dstImg, err = converter.Convert(ctx, client, dstRef, srcRef, opts...)
fmt.Println(dstImg.Target)
```
ctr example: `ctr images convert --oci --uncompress example.com/foo:orig example.com/foo:converted`
Go test: `go test -exec sudo -test.root -test.run TestConvert`
The implementation is from https://github.com/containerd/stargz-snapshotter/pull/224,
but eStargz-specific functions are not included in this PR.
eStargz converter can be specified by importing `estargz` package and using `WithLayerConvertFunc(estargz.LayerConvertFunc)` option.
This converter interface will be potentially useful for converting zstd and ocicrypt layers as well.
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
If we print message when SIG_PIPE occuers in signal handler.
There is a loop {print->SIG_PIPE->print->SIG_PIPE...}, which consume
a lot of cpu time. So do not print message in this situaiton.
Signed-off-by: Liu Hua <weldonliu@tencent.com>
The additional []containerd.RemoteOpt is not used by ctr currently,
but planned to be used by nerdctl and also probably by stargz's ctr-remote.
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
Go 1.14 introduced a change to os.OpenFile (and syscall.Open) on Windows
that uses the permissions passed to determine if the file should be
created read-only or not. If the user-write bit (0200) is not set, then
FILE_ATTRIBUTE_READONLY is set on the underlying CreateFile call.
This is a significant change for any Windows code which created new
files and set the permissions to 0 (previously the permissions had no
affect, so some code didn't set them at all).
This change fixes the issue for the Windows service panic file. It will
now properly be created as a non-read-only file on Go 1.14+.
I have looked over the rest of the containerd code and didn't see other
places where this seems like an issue.
Signed-off-by: Kevin Parsons <kevpar@microsoft.com>
This allows filesystem-based ACLs for configuring access to the socket
of a shim.
Ported from Michael Crosby's similar patch for v2 shims.
Signed-off-by: Samuel Karp <skarp@amazon.com>
This allows filesystem based ACLs for configuring access to the socket of a
shim.
Co-authored-by: Samuel Karp <skarp@amazon.com>
Signed-off-by: Samuel Karp <skarp@amazon.com>
Signed-off-by: Michael Crosby <michael@thepasture.io>
Signed-off-by: Michael Crosby <michael.crosby@apple.com>
This adds linux cni support to `ctr run` via a `--cni` flag. This uses the
default configuration for CNI on `ctr` to configure the network namespace for a
container.
Signed-off-by: Michael Crosby <michael@thepasture.io>
Starting with go1.14, the go runtime hijacks SIGURG but with no way to
not send to other signal handlers.
In practice, we get this signal frequently.
I found this while testing out go1.15 with ctr and multiple execs with
only `echo hello`. When the process exits quickly, if the previous
commit is not applied, you end up with an error message that it couldn't
forward SIGURG to the container (due to the process being gone).
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Previously the signal loop can end up racing with the process exiting.
Intead of logging and continuing the loop, exit early.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Put the overlay plugin in a separate package to allow the overlay package to be
used without needing to import and initialize the plugin.
Signed-off-by: Derek McGowan <derek@mcg.dev>
The flag was being silently ignored, and so no network setup was done,
and no notice given.
Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
We should never return a nil context because of the way this function is
typically used... e.g.
```
ctx, done, err := containerd.WithLease(ctx)
```
If there is an error `ctx` will be nil and any error handling may cause
an NPE if it tries to use `ctx`.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
I missed this in my previous change: the ctr man page is also
in Section 8, because it's considered an administrative tool,
and containerd-config is related to containerd so updating these
as well.
This commit also fixes naming of the generated files, which was
hard-coded to .1.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Moved registry host configuration to the config package
and allows support of loading configurations from a
directory when the hosts are being resolved.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Add `remotes/certutil` functions for loading `ca.crt`, `client.cert`, and `client.key` into `tls.Config` from a directory like `/etc/docker/certs.d/<hostname>.
See https://docs.docker.com/engine/security/certificates/ .
Client applications including CRI plugin are expected to configure the resolver using these functions.
As an example, the `ctr` tool is extended to support `ctr images pull --certs-dir=/etc/docker/certs.d example.com/foo/bar:baz`.
Tested with Harbor 1.8.
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
If runtime is not runc, it doesn't make sense to send runc Options
as container create options, which will confuse other runtimes and
it actually causes kata shimv2 to fail to unmarshal the requset.
Signed-off-by: Peng Tao <bergwolf@hyper.sh>
The builtins_cri_linux.go file only being included for _linux builds which
means that the builds for Windows do not contain the CRI plugin. This can
be disabled for either platform with the no_cri tag.
Signed-off-by: Justin Terry (SF) <juterry@microsoft.com>
Errors like `"type with url %s: not found"` are typical for non-builtin
event types, and should not resultin exiting `ctr`.
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
* --uidmap support for one remapping
* --gidmap support for one remapping
* create IoUid and IoGid options for getNewTaskOpts
Signed-off-by: Jie Hao Liao <liaojh1998@gmail.com>
* only shim v2 runc v2 ("io.containerd.runc.v2") is supported
* only PID metrics is implemented. Others should be implemented in separate PRs.
* lots of code duplication in v1 metrics and v2 metrics. Dedupe should be separate PR.
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
This prepares us to be able to update docker/docker vendoring to a
recent commit.
Signed-off-by: Phil Estes <estesp@linux.vnet.ibm.com>
Signed-off-by: Phil Estes <estesp@linux.vnet.ibm.com>
Seems to be that docs/man/ctr.1.md and docs/man/containerd.1.md were
removed in #3637 and were not updated correctly in the Makefile, leading
to build failures like:
+ make man
make: *** No rule to make target `man/ctr.1', needed by `man'. Stop.
Changes the gen-manpages command to be specific on which manpages are to
be generated.
Signed-off-by: Eli Uriegas <eli.uriegas@docker.com>
This change includes a cri master bump and a cgroup bump for windows support
with cgroup stats and reusing the cgroup metric types.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
The climan package has a command that can be registered with any urfav
cli app to generate man pages.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
The cgroup dependency brings in quite a lot only for WithNamespaceCgroupDeletion,
which is a namespaces.DeleteOpt.
Signed-off-by: Tibor Vass <tibor@docker.com>
- Add `all-manifests` option to both `ctr content fetch` and `ctr
images pull`. By default it is false.
- This option ties to `AppendDistributionSourceLabel` in client.
Signed-off-by: Yu Yi <yiyu@google.com>
Previously the TTRPC address was generated as "<GRPC address>.ttrpc".
This change now allows explicit configuration of the TTRPC address, with
the default still being the old format if no value is specified.
As part of this change, a new configuration section is added for TTRPC
listener options.
Signed-off-by: Kevin Parsons <kevpar@microsoft.com>
EventLog is very old and provides a poor experience. We have supported
ETW for logging for a while, which is much better. We have also
observed an issue where EventLog keeps containerd.exe open, preventing
containerd from being upgraded to a new version. Due to all of this,
it makes sense to remove the old EventLog hook in favor of using ETW
logging on Windows as the primary diagnostic experience.
Signed-off-by: Kevin Parsons <kevpar@microsoft.com>
This adds a singleton `timeout` package that will allow services and user
to configure timeouts in the daemon. When a service wants to use a
timeout, it should declare a const and register it's default value
inside an `init()` function for that package. When the default config
is generated, we can use the `timeout` package to provide the available
timeout keys so that a user knows that they can configure.
These show up in the config as follows:
```toml
[timeouts]
"io.containerd.timeout.shim.cleanup" = 5
"io.containerd.timeout.shim.load" = 5
"io.containerd.timeout.shim.shutdown" = 3
"io.containerd.timeout.task.state" = 2
```
Timeouts in the config are specified in seconds.
Timeouts are very hard to get right and giving this power to the user to
configure things is a huge improvement. Machines can be faster and
slower and depending on the CPU or load of the machine, a timeout may
need to be adjusted.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
We are separating out the encryption code and have designed a few new
interfaces and APIs for processing content streams. This keep the core
clean of encryption code but enables not only encryption but support of
multiple content types ( custom media types ).
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Closes#1862
This adds a new rpc to the introspection service to provide server
information with a generated UUID that is done on demand and the os and
arch of the server.
ctr output:
```bash
> sudo ctr version
Client:
Version: v1.2.0-802-g57821695.m
Revision: 578216950de9c1c188708369e2a31ac6c494dfee.m
Server:
Version: v1.2.0-802-g57821695.m
Revision: 578216950de9c1c188708369e2a31ac6c494dfee.m
UUID: 92e982a9-f13e-4a2c-9032-e69b27fed454
OS: linux
ARCH: amd64
```
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Creating a console for containerd causes it to be inherited by any child
processes, which gives us performance and reliability improvements. See
comment in code for more information.
Another option considered here would be to invoke each child process
with the DETACHED_PROCESS flag. This would save us the containerd
console allocation. The difficulty of this approach would be ensuring
that all process invocation points have had this flag added, and that
any future invocations also use the flag.
Signed-off-by: Kevin Parsons <kevpar@microsoft.com>
This change moves from specific, global errors to the errdefs errors.
This makes it easy to handle certain classes of errors while still
adding context to the failure.
Signed-off-by: Stephen Day <stephen.day@getcruise.com>
Adds a subcommand for tagging images managed by containerd.
Usage:
ctr images tag docker.io/library/alpine:latest docker.io/library/alpine:new
Comes with a --force flag to allow you to gracefully overwrite image references
that already exist.
Signed-off-by: Eli Uriegas <eli.uriegas@docker.com>
This also refactors the lcow and windows
snapshotters to use go-winio's utility functions for checking the
filesystem type.
Signed-off-by: Eric Hotinger <ehotinger@gmail.com>
Closes#3305
This adds an opt and a `--cgroup,-c` flag to `ctr namespaces rm` to
remove the cgroup that is commonly created with runtimes.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Previously we waited for 60 seconds after the service faults to restart
it. However, there isn't much benefit to waiting this long. We expect
15 seconds to be a more reasonable delay.
Signed-off-by: Kevin Parsons <kevpar@microsoft.com>
make ctr shim command easy to use for user, shim socket is generated
through sha256, and it can not get directly, change socket flag to id
command, generated socket in code.
It also avoid fail to connect shim v2, since shim v2 have multiple
containers, `ctr shim --socket state` should specify container id, or
get error `rpc error: code = NotFound desc = container not created: not
found`
Signed-off-by: Ace-Tang <aceapril@126.com>
Closes#603
This adds logging facilities at the shim level to provide minimal I/O
overhead and pluggable logging options. Log handling is done within the
shim so that all I/O, cpu, and memory can be charged to the container.
A sample logging driver setting up logging for a container the systemd
journal looks like this:
```go
package main
import (
"bufio"
"context"
"fmt"
"io"
"sync"
"github.com/containerd/containerd/runtime/v2/logging"
"github.com/coreos/go-systemd/journal"
)
func main() {
logging.Run(log)
}
func log(ctx context.Context, config *logging.Config, ready func() error) error {
// construct any log metadata for the container
vars := map[string]string{
"SYSLOG_IDENTIFIER": fmt.Sprintf("%s:%s", config.Namespace, config.ID),
}
var wg sync.WaitGroup
wg.Add(2)
// forward both stdout and stderr to the journal
go copy(&wg, config.Stdout, journal.PriInfo, vars)
go copy(&wg, config.Stderr, journal.PriErr, vars)
// signal that we are ready and setup for the container to be started
if err := ready(); err != nil {
return err
}
wg.Wait()
return nil
}
func copy(wg *sync.WaitGroup, r io.Reader, pri journal.Priority, vars map[string]string) {
defer wg.Done()
s := bufio.NewScanner(r)
for s.Scan() {
if s.Err() != nil {
return
}
journal.Send(s.Text(), pri, vars)
}
}
```
A `logging` package has been created to assist log developers create
logging plugins for containerd.
This uses a URI based approach for logging drivers that can be expanded
in the future.
Supported URI scheme's are:
* binary
* fifo
* file
You can pass the log url via ctr on the command line:
```bash
> ctr run --rm --runtime io.containerd.runc.v2 --log-uri binary://shim-journald docker.io/library/redis:alpine redis
```
```bash
> journalctl -f -t default:redis
-- Logs begin at Tue 2018-12-11 16:29:51 EST. --
Mar 08 16:08:22 deathstar default:redis[120760]: 1:C 08 Mar 2019 21:08:22.703 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
Mar 08 16:08:22 deathstar default:redis[120760]: 1:M 08 Mar 2019 21:08:22.704 # You requested maxclients of 10000 requiring at least 10032 max file descriptors.
Mar 08 16:08:22 deathstar default:redis[120760]: 1:M 08 Mar 2019 21:08:22.704 # Server can't set maximum open files to 10032 because of OS error: Operation not permitted.
Mar 08 16:08:22 deathstar default:redis[120760]: 1:M 08 Mar 2019 21:08:22.704 # Current maximum open files is 1024. maxclients has been reduced to 992 to compensate for low ulimit. If you need higher maxclients increase 'ulimit -n'.
Mar 08 16:08:22 deathstar default:redis[120760]: 1:M 08 Mar 2019 21:08:22.705 * Running mode=standalone, port=6379.
Mar 08 16:08:22 deathstar default:redis[120760]: 1:M 08 Mar 2019 21:08:22.705 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
Mar 08 16:08:22 deathstar default:redis[120760]: 1:M 08 Mar 2019 21:08:22.705 # Server initialized
Mar 08 16:08:22 deathstar default:redis[120760]: 1:M 08 Mar 2019 21:08:22.705 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
Mar 08 16:08:22 deathstar default:redis[120760]: 1:M 08 Mar 2019 21:08:22.705 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
Mar 08 16:08:22 deathstar default:redis[120760]: 1:M 08 Mar 2019 21:08:22.705 * Ready to accept connections
Mar 08 16:08:50 deathstar default:redis[120760]: 1:signal-handler (1552079330) Received SIGINT scheduling shutdown...
Mar 08 16:08:50 deathstar default:redis[120760]: 1:M 08 Mar 2019 21:08:50.405 # User requested shutdown...
Mar 08 16:08:50 deathstar default:redis[120760]: 1:M 08 Mar 2019 21:08:50.406 * Saving the final RDB snapshot before exiting.
Mar 08 16:08:50 deathstar default:redis[120760]: 1:M 08 Mar 2019 21:08:50.452 * DB saved on disk
Mar 08 16:08:50 deathstar default:redis[120760]: 1:M 08 Mar 2019 21:08:50.453 # Redis is now ready to exit, bye bye...
```
The following client side Opts are added:
```go
// LogURI provides the raw logging URI
func LogURI(uri *url.URL) Creator { }
// BinaryIO forwards contianer STDOUT|STDERR directly to a logging binary
func BinaryIO(binary string, args map[string]string) Creator {}
```
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
We can use cross repository push feature to reuse the existing blobs in
the same registry. Before make push fast, we know where the blob comes
from.
Use the `containerd.io/distribution.source. = [,]` as label format. For
example, the blob is downloaded by the docker.io/library/busybox:latest
and the label will be
containerd.io/distribution.source.docker.io = library/busybox
If the blob is shared by different repos in the same registry, the repo
name will be appended, like:
containerd.io/distribution.source.docker.io = library/busybox,x/y
NOTE:
1. no need to apply for legacy docker image schema1.
2. the concurrent fetch actions might miss some repo names in label, but
it is ok.
3. it is optional. no need to add label if the engine only uses images
not push.
Signed-off-by: Wei Fu <fuweid89@gmail.com>
If user sets debug mode, the command push should only show the debug log
information. If the stdout is with flush by the progress status, it is
hard to see the debug log.
Signed-off-by: Wei Fu <fuweid89@gmail.com>
Signed-off-by: John Howard <jhoward@microsoft.com>
Needed for the containerd work on Windows and integrating the
oci package from containerd into moby.
No longer sets defaults for
- .Process.ConsoleSize
- .Windows.IgnoreFlushesDuringBoot
- .Windows.Network.AllowUnqualifiedDNSQuery
Adds helper functions and tests for
- WithWindowsIgnoreFlushesDuringBoot
- WithWindowNetworksAllowUnqualifiedDNSQuery
Updates `ctr run` on Windows to use the new helper functions,
ConsoleSize is already handled.
megacheck, gosimple and unused has been deprecated and subsumed by
staticcheck. And staticcheck also has been upgraded. we need to update
code for the linter issue.
close: #2945
Signed-off-by: Wei Fu <fuweid89@gmail.com>
This brings freebsd in line with Darwin, ie it builds, but some parts may not yet
be fully functional. There is now a WIP `runc` port for FreeBSD at
https://github.com/clovertrail/runc/tree/1501-SupportOnFreeBSD so should be able
to test further.
Signed-off-by: Justin Cormack <justin@specialbusservice.com>
The $EDITOR is not clear for user. If the env doesn't set the value for
$EDITOR, the users don't know how to use this subcommand when they get
error like this:
```
sh: 1: /tmp/edit-605573012: Permission denied
```
Signed-off-by: Wei Fu <fuweid89@gmail.com>
Adds ctr run --memory-limit for all platforms.
Adds ctr run --cpu-count for Windows platforms.
Signed-off-by: Justin Terry (VM) <juterry@microsoft.com>
support checkpoint without committing a checkpoint dir into a
checkpoint image and restore without untar image into checkpoint
directory. support for both v1 and v2 runtime
Signed-off-by: Ace-Tang <aceapril@126.com>
Signed-off-by: John Howard <jhoward@microsoft.com>
Allows containerd.exe to run as a Windows service. eg
Register: `.\containerd.exe --register-service`
Start: `net start containerd`
...
Stop: `net stop containerd`
Unregister: `.\containerd.exe --unregister-service`
When running as a service, logs will go to the Windows application
event log.
when use `ctr t checkpoint`, runtime name should get from container
info, but not passed by flag, since task has already running with
specified runtime
Signed-off-by: Ace-Tang <aceapril@126.com>
