Commit Graph

2903 Commits

Author SHA1 Message Date
Stephen Day
551579b316 Merge pull request #1621 from dmcgowan/refactor-diff
Refactor differ into separate package
2017-10-11 10:21:19 -07:00
Derek McGowan
64a96aab8a
Abort newly created references on error
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-10-11 10:10:19 -07:00
Derek McGowan
69e7e7fa4f
Add error wrapping for client checkpoint errors
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-10-11 10:02:29 -07:00
Derek McGowan
d9db1d112d
Refactor differ into separate package
Add differ options and package with interface.
Update optional values on diff interface to use options.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-10-11 10:02:29 -07:00
Michael Crosby
60960e1c17 Merge pull request #1597 from stevvooe/introspection-service
api/services: define the introspection API
2017-10-11 10:57:11 -04:00
Stephen J Day
0e72ce5c5b
services/introspection: implement plugin reporting
With this change, we integrate all the plugin changes into the
introspection service.

All plugins can be listed with the following command:

```console
$ ctr plugins
TYPE                            ID             PLATFORM       STATUS
io.containerd.content.v1        content        -              ok
io.containerd.metadata.v1       bolt           -              ok
io.containerd.differ.v1         walking        linux/amd64    ok
io.containerd.grpc.v1           containers     -              ok
io.containerd.grpc.v1           content        -              ok
io.containerd.grpc.v1           diff           -              ok
io.containerd.grpc.v1           events         -              ok
io.containerd.grpc.v1           healthcheck    -              ok
io.containerd.grpc.v1           images         -              ok
io.containerd.grpc.v1           namespaces     -              ok
io.containerd.snapshotter.v1    btrfs          linux/amd64    error
io.containerd.snapshotter.v1    overlayfs      linux/amd64    ok
io.containerd.grpc.v1           snapshots      -              ok
io.containerd.monitor.v1        cgroups        linux/amd64    ok
io.containerd.runtime.v1        linux          linux/amd64    ok
io.containerd.grpc.v1           tasks          -              ok
io.containerd.grpc.v1           version        -              ok
```

There are few things to note about this output. The first is that it is
printed in the order in which plugins are initialized. This useful for
debugging plugin initialization problems. Also note that even though the
introspection GPRC api is a itself a plugin, it is not listed. This is
because the plugin takes a snapshot of the initialization state at the
end of the plugin init process. This allows us to see errors from each
plugin, as they happen. If it is required to introspect the existence of
the introspection service, we can make modifications to include it in
the future.

The last thing to note is that the btrfs plugin is in an error state.
This is a common state for containerd because even though we load the
plugin, most installations aren't on top of btrfs and the plugin cannot
be used. We can actually view this error using the detailed view with a
filter:

```console
$ ctr plugins --detailed id==btrfs
Type:          io.containerd.snapshotter.v1
ID:            btrfs
Platforms:     linux/amd64
Exports:
               root      /var/lib/containerd/io.containerd.snapshotter.v1.btrfs
Error:
               Code:        Unknown
               Message:     path /var/lib/containerd/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter
```

Along with several other values, this is a valuable tool for evaluating the
state of components in containerd.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-10-10 17:29:24 -07:00
Stephen J Day
49814646cf
plugin: allow declaring a plugin depends on all others
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-10-10 16:40:47 -07:00
Stephen J Day
8508e8252b
plugin: refactor plugin system to support initialization reporting
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-10-10 16:40:47 -07:00
Stephen J Day
fe52d9369f
errdefs: use status package for mapping errors
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-10-10 16:40:47 -07:00
Stephen J Day
d8bd6b70ce
plugin: allow querying plugin graph to be re-entrant
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-10-10 16:40:47 -07:00
Stephen J Day
363d692f35
api/services: define the introspection API
After several requests for information about platform support,
configuration introspection and feature switches, we now have a solution
that should work in all these use cases. The Introspection API hooks
into the plugin subsystem registration process. During registration,
plugins declare several pieces of information, allowing clients to
discover the cababilities and support that a containerd instance
provides, including whether or not it loaded with an error.

To allow symmetrical error reporting, the `google/rpc.Status` protobuf
definitions have been brought in from the googleapis project.
Unfortunately, we had to generate these in place to match our protobuf
system.

Once we like this design, we can add an implementation to integrate it
directly with the plugin system.

Enjoy!

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-10-10 16:40:47 -07:00
Michael Crosby
12c79cc0d9 Merge pull request #1604 from jessvalarezo/windows-listpids
ListPids returns process details in addition to process IDs.
2017-10-10 19:26:47 -04:00
Jess
061c719209 ListPids returns process ID and other info
Signed-off-by: Jess <jessica.valarezo@docker.com>
2017-10-10 22:57:15 +00:00
Phil Estes
96725a5a0b Merge pull request #1620 from mlaventure/runc-io-id
Allow setting the uid & gid of the io pipes
2017-10-10 17:12:34 -04:00
Kenfe-Mickael Laventure
c807ba8379
Allow setting the uid & gid of the io pipes
Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
2017-10-10 13:24:12 -07:00
Phil Estes
1ea8ac41d3 Merge pull request #1617 from crosbymichael/2cgroup
Use namespace in default cgroup path
2017-10-10 10:55:12 -04:00
Michael Crosby
d7864eb77b Use namespace in default cgroup path
By default, the generated spec will place containers in cgroups by their
ids, we need to use the namespace as the cgroup root to avoid
containers with the same name being placed in the same cgroup.

```
11:perf_event:/to/redis
10:freezer:/to/redis
9:memory:/to/redis
8:devices:/to/redis
7:net_cls,net_prio:/to/redis
6:pids:/to/redis
5:hugetlb:/to/redis
4:cpuset:/to/redis
3:blkio:/to/redis
2:cpu,cpuacct:/to/redis
1:name=systemd:/to/redis

11:perf_event:/te/redis
10:freezer:/te/redis
9:memory:/te/redis
8:devices:/te/redis
7:net_cls,net_prio:/te/redis
6:pids:/te/redis
5:hugetlb:/te/redis
4:cpuset:/te/redis
3:blkio:/te/redis
2:cpu,cpuacct:/te/redis
1:name=systemd:/te/redis
```

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-10-09 17:07:57 -04:00
Michael Crosby
72bb45ac46 Merge pull request #1603 from mlaventure/fix-windows-flaky-tests
windows: ensure a init process exist after "Create()"
2017-10-09 11:58:23 -04:00
Kenfe-Mickael Laventure
cfa87567a0
windows: Create init process with task
This prevents `task.Wait()` to return an error if it is called before the task
is started.

Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
2017-10-09 08:40:12 -07:00
Kenfe-Mickael Laventure
ad5266456c
windows: Fix a few races
Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
2017-10-09 08:40:12 -07:00
Kenfe-Mickaël Laventure
b2e348231c Merge pull request #1584 from miaoyq/fix-mount-lookup-err
Fixes looking up mountinfo corresponds to path
2017-10-09 08:35:01 -07:00
Yanqiang Miao
d7c4611848 Ensure getting the correct mountinfo corresponds to path
Signed-off-by: Yanqiang Miao <miao.yanqiang@zte.com.cn>
2017-10-08 17:16:28 +08:00
Akihiro Suda
a8426ed9e7 Merge pull request #1609 from crosbymichael/attach-docs
Add docs around multiple attach
2017-10-08 11:24:57 +09:00
Derek McGowan
6c3f393765 Merge pull request #1611 from stevvooe/address-cgroup-merge-conflicty
metrics/cgroups: handle error on call to cgroup
2017-10-06 16:09:20 -07:00
Stephen J Day
77e5f6553c
metrics/cgroups: handle error on call to cgroup
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-10-06 15:51:20 -07:00
Michael Crosby
bbb01ee1ab Add docs around multiple attach
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-10-06 17:31:11 -04:00
Phil Estes
987fcd1201 Merge pull request #1598 from Random-Liu/fix-load-task
Fix task load.
2017-10-06 16:38:40 -04:00
Stephen Day
67bbfc5d1c Merge pull request #1605 from dmcgowan/metadata-db-documentation
Add documentation for metadata database
2017-10-06 13:37:00 -07:00
Derek McGowan
e4defbcb6d
Add documentation for metadata database
Add documentation and explanation for migrations.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-10-06 13:33:07 -07:00
Michael Crosby
6f351f0a85 Merge pull request #1586 from mathieui/postmortem-stats
Collect cgroup stats one last time before exit
2017-10-06 15:44:28 -04:00
Michael Crosby
d92f6eea1f Allow blocking and non-blocking metrics collection
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-10-06 12:05:56 -04:00
Mathieu Pasquet
ed519bb5ce Collect cgroup stats one last time before exit
This commit adds a collection step in the Stop() task handler which will
retrieve the metrics available for this container at that time, and
store them until the next prometheus Collect() cycle.

This allows short-lived containers to be visible in prometheus, which
would otherwise be ignored (for example, running containerd-stress would
show something like 2 or 3 containers in the end, while now we can see
all of them). It also allows for more accurate collection when
long-running containers end (for example CPU usage could spike in the
last few seconds).

A simple case illustrating this with cpu usage would be:

  ctr run -t --rm docker.io/library/alpine:latest mycontainer sh -c 'yes > /dev/null & sleep 3 && pkill yes'

Signed-off-by: Mathieu Pasquet <mathieu.pasquet@alterway.fr>
2017-10-06 15:40:38 +02:00
Stephen Day
8558b98eb1 Merge pull request #1582 from dmcgowan/metadata-db-object
Create metadata db object
2017-10-05 16:50:44 -07:00
Derek McGowan
7f657ce3de
Add database migrations
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-10-05 16:35:58 -07:00
Lantao Liu
28ca8f05d3 Fix task load.
Signed-off-by: Lantao Liu <lantaol@google.com>
2017-10-05 21:03:24 +00:00
Phil Estes
7c4bca591f Merge pull request #1600 from AkihiroSuda/defaultspec
spec: set MaskedPaths and ReadOnlyPaths by default
2017-10-05 12:16:09 -04:00
Akihiro Suda
d7b0e522c7 spec: set MaskedPaths and ReadOnlyPaths by default
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-10-05 06:01:35 +00:00
Phil Estes
72a3a019ae Merge pull request #1595 from crosbymichael/container-lock
sync container metadata before operations
2017-10-04 21:48:30 -04:00
Stephen Day
28882fddae Merge pull request #1589 from mlaventure/shim-no-newns
Add an option to prevent putting the shim in a new mount namespace
2017-10-04 17:24:26 -07:00
Michael Crosby
fa9e9bdf46 Fetch current container info before operations
This makes sure the client is always in sync with the server before
performing any type of operations on the container metadata.

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-10-04 15:29:57 -04:00
Derek McGowan
8d892a651b
Update metadata plugin initialization
Updates metadata plugin to require content and
snapshotter plugins be loaded and initializes with
those plugins, keeping the metadata database structure
static after initialization. Service plugins now only
require metadata plugin access snapshotter or content
stores through metadata, which was already required
behavior of the services.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-10-04 11:25:46 -07:00
Derek McGowan
2ab70f21ac
Add content and snapshot store references
Update database object to hold reference to the
data stores.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-10-04 11:16:22 -07:00
Derek McGowan
447a0a9452
Add children bucket to back reference snapshots
Adds back links from parent to children in order to prevent
deletion of a referenced snapshot in a namespace.
Avoid removing snapshot during metadata delete to
prevent shared namespaces from being mistakenly deleted.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-10-04 11:16:22 -07:00
Derek McGowan
56c1b79a4c
Create metadata db object
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-10-04 11:16:22 -07:00
Kenfe-Mickaël Laventure
acba0f50ef Merge pull request #1594 from akimd/akim/clean
style: move definition for consistency
2017-10-04 09:18:43 -07:00
Kenfe-Mickaël Laventure
ec43dc2b46 Merge pull request #1592 from AkihiroSuda/workdir2
ctr run: add --cwd
2017-10-04 09:17:21 -07:00
Kenfe-Mickael Laventure
26d4c2c217
Add an option to prevent putting the shim in a new mount namespace
This is needed for users on kernel older than 3.18 so they can avoid EBUSY
errors when trying to unlink, rename or remove a mountpoint that is present in
a shim namespace.

Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
2017-10-04 09:16:02 -07:00
Michael Crosby
0ae9d8fabf Merge pull request #1567 from stevvooe/image-check-function
images: support checking status of image content
2017-10-04 11:40:43 -04:00
Akim Demaille
8ceeeed4ae style: move definition for consistency
Signed-off-by: Akim Demaille <akim.demaille@docker.com>
2017-10-04 17:02:51 +02:00
Akihiro Suda
b26e2e781c ctr run: add --cwd
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-10-04 04:40:08 +00:00