Commit Graph

488 Commits

Author SHA1 Message Date
Akihiro Suda
04b98bb0ee
Deprecate legacy shims
Fix #4365

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2020-07-30 01:57:12 +09:00
Sebastiaan van Stijn
f2edc6f164
vendor: update gotest.tools v3.0.2
full diff: https://github.com/gotestyourself/gotest.tools/compare/v2.3.0...v3.0.2

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-02-28 17:47:20 +01:00
Akihiro Suda
8e448bb279 vendor protobuf & grpc
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2020-02-26 10:57:05 +09:00
Phil Estes
7804afb226
Merge pull request #3950 from dmcgowan/fix-printf-scanner-error
Fix filter errors
2020-01-09 09:21:30 -05:00
Derek McGowan
3af3a76026
Fix filter errors
Prevent error messages from being output to stderr.
Return illegal token when a quoted string is invalid and
capture the error.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2020-01-08 18:07:50 -08:00
Phil Estes
80d50a8a63
Merge pull request #3925 from dmcgowan/snapshot-cleanup-api
Add Cleanup to snapshot API
2020-01-08 10:03:18 -05:00
Derek McGowan
123af61c0b
Add Cleanup to snapshot API
Cleanup is an optional method a snapshotter may implement.
Cleanup can be used to cleanup resources after a snapshot
has been removed. This function allows a snapshotter to defer
longer resource cleanup until after snapshot removals are
completed. Adding this to the API allows proxy snapshotters
to leverage this enhancement.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2020-01-07 14:59:20 -08:00
Kathryn Baldauf
a18f77bea0 create local version of introspection service
Signed-off-by: Kathryn Baldauf <kabaldau@microsoft.com>
2020-01-02 12:34:23 -08:00
Akihiro Suda
8f870c233f support cgroup2
* only shim v2 runc v2 ("io.containerd.runc.v2") is supported
* only PID metrics is implemented. Others should be implemented in separate PRs.
* lots of code duplication in v1 metrics and v2 metrics. Dedupe should be separate PR.

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2019-12-12 02:56:51 +09:00
Michael Crosby
f9d8f0e56e
Merge pull request #3794 from fahedouch/server_test
add tests to server_test
2019-11-04 10:37:26 -05:00
fahedouch
7c016b655b
add tests to server_test
Signed-off-by: fahedouch <fahed.dorgaa@gmail.com>
2019-11-04 09:15:05 -05:00
Ameya Gawde
d21f0f116e
windows process shim installer
Signed-off-by: Ameya Gawde <ameya.gawde@docker.com>
2019-10-30 16:22:02 -07:00
Derek McGowan
66aa1d3ef6
Add snapshot walk implementations
Temporarily remove zfs and aufs until interface update

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2019-10-24 11:11:22 -07:00
Lantao Liu
aaccfcbe2b Fix containerd config dump.
Signed-off-by: Lantao Liu <lantaol@google.com>
2019-10-23 22:38:18 -07:00
Lantao Liu
ffcb1cc9be Fix delete error code on the containerd daemon side.
Signed-off-by: Lantao Liu <lantaol@google.com>
2019-10-09 00:28:51 -07:00
Derek McGowan
0b224ac7d6
Update metadata interfaces for containers and leases
Add more thorough dirty checking across all types which
may be deleted and hold references.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2019-09-23 15:27:39 -07:00
Wei Fu
3d1fdacccf config_test: sort result before check
config imports is in random order after LoadConfig call so that the
testing check should be done after sort.Strings(out.Imports).

Signed-off-by: Wei Fu <fuweid89@gmail.com>
2019-09-05 16:30:28 +08:00
Michael Crosby
f76eefd272
Merge pull request #3574 from mxpv/cfg
Support config imports
2019-09-04 16:34:11 -04:00
Derek McGowan
a4482d9a6f
Fix proxy plugin config validation
Proxy plugins are keyed only on the identifier, the type
is specified within the proxy plugin configuration which
maps to the full URI. The proxy plugin configuration is
not passed to the plugin for configuration like other
plugins.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2019-09-03 17:31:14 -07:00
Maksym Pavlenko
24b9e2c1a0 Merge configs section by section
Signed-off-by: Maksym Pavlenko <makpav@amazon.com>
2019-08-23 15:49:02 -07:00
Maksym Pavlenko
8ebffecbc3 Use map for stream processors
Signed-off-by: Maksym Pavlenko <makpav@amazon.com>
2019-08-23 15:31:37 -07:00
Maksym Pavlenko
19cd0a4f12 Append slices when importing config files
Signed-off-by: Maksym Pavlenko <makpav@amazon.com>
2019-08-23 10:12:19 -07:00
Maksym Pavlenko
bca0857530 Fix toml plugin decoding
Do not rely on toml metadata when decoding plugin's configs as it's not possible to merge toml.MetaData structs during import.

Signed-off-by: Maksym Pavlenko <makpav@amazon.com>
2019-08-22 17:22:25 -07:00
Maksym Pavlenko
a1e3779cad Support config imports #3289
Signed-off-by: Maksym Pavlenko <makpav@amazon.com>
2019-08-22 15:50:58 -07:00
Kevin Parsons
d7e1b25384 Allow explicit configuration of TTRPC address
Previously the TTRPC address was generated as "<GRPC address>.ttrpc".
This change now allows explicit configuration of the TTRPC address, with
the default still being the old format if no value is specified.

As part of this change, a new configuration section is added for TTRPC
listener options.

Signed-off-by: Kevin Parsons <kevpar@microsoft.com>
2019-08-22 00:56:27 -07:00
Michael Crosby
2e8ea9fd6b Allow timeouts to be configured in config
This adds a singleton `timeout` package that will allow services and user
to configure timeouts in the daemon.  When a service wants to use a
timeout, it should declare a const and register it's default value
inside an `init()` function for that package.  When the default config
is generated, we can use the `timeout` package to provide the available
timeout keys so that a user knows that they can configure.

These show up in the config as follows:

```toml
[timeouts]
  "io.containerd.timeout.shim.cleanup" = 5
  "io.containerd.timeout.shim.load" = 5
  "io.containerd.timeout.shim.shutdown" = 3
  "io.containerd.timeout.task.state" = 2

```

Timeouts in the config are specified in seconds.

Timeouts are very hard to get right and giving this power to the user to
configure things is a huge improvement.  Machines can be faster and
slower and depending on the CPU or load of the machine, a timeout may
need to be adjusted.

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2019-08-13 17:36:32 +00:00
Michael Crosby
f867401c69 Use fds and pass Payloads over diff api
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2019-08-07 18:35:55 +00:00
Michael Crosby
366823727f Add server config for stream processors
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2019-08-07 18:35:55 +00:00
Michael Crosby
9f0bc2b53f Add UUID and server info to introspection
Closes #1862

This adds a new rpc to the introspection service to provide server
information with a generated UUID that is done on demand and the os and
arch of the server.

ctr output:

```bash
> sudo ctr version

Client:
  Version:  v1.2.0-802-g57821695.m
  Revision: 578216950de9c1c188708369e2a31ac6c494dfee.m

Server:
  Version:  v1.2.0-802-g57821695.m
  Revision: 578216950de9c1c188708369e2a31ac6c494dfee.m
  UUID: 92e982a9-f13e-4a2c-9032-e69b27fed454
  OS: linux
  ARCH: amd64
```

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2019-08-06 14:40:48 +00:00
Lantao Liu
bb99688914 Fix containerd on windows.
Signed-off-by: Lantao Liu <lantaol@google.com>
2019-08-02 23:18:35 -07:00
Michael Crosby
eb4b3e8772 Fast path getting pid from task
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2019-07-26 17:48:00 +00:00
Michael Crosby
101d4b78eb Add timeout to task state calls
Fixes #3440

This also returns the task that times out or has an error on the state
call in an UNKNOWN status so that the user can manually kill and remove
the task.

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2019-07-23 19:58:41 +00:00
Maksym Pavlenko
ef7f46eb7b Fix linter errors
Signed-off-by: Maksym Pavlenko <makpav@amazon.com>
2019-07-14 20:49:40 -07:00
Michael Crosby
9547d269a1 Add v2 server config support with plugin URIs
Closes #3210

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2019-06-10 17:54:20 +00:00
Michael Crosby
d4e7efbb93
Merge pull request #3304 from fuweid/me-update-lease
leases: support resource management
2019-06-03 16:43:34 -04:00
Michael Crosby
2ec2089b05
Merge pull request #3123 from stefanberger/extend_apply_call_with_options_parameter
Extend Applier's Apply() method with an optional options parameter
2019-06-03 11:13:40 -04:00
Wei Fu
8a388d6238 leases: support resource management
Add three methods for lease service so that the client can use it to
manage the resource by lease, not just gc.root label. With the following
methods, it is easy for client to maintain their own cache system.

```
 - AddResource(context.Context, Lease, Resource) error
 - RemoveResource(context.Context, Lease, Resource) error
 - ListResources(context.Context, Lease) ([]Resource, error)
```

And the resource is to be

```golang
type Resource {
  ID   string
  Type string
}
```

For the snapshots, the Type field will be formatted by
snapshots/%{type}, like snapshots/overlayfs.

fix: #3295

Signed-off-by: Wei Fu <fuweid89@gmail.com>
2019-05-29 11:00:32 +08:00
Sebastiaan van Stijn
2ed8e60fa1
bump google.golang.org/grpc v1.20.1
full diff: https://github.com/grpc/grpc-go/compare/v1.12.2...v1.20.1

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-05-05 12:39:23 -07:00
Davor Kapsa
cfc36388b3 Remove redundant error checks
Signed-off-by: Davor Kapsa <davor.kapsa@gmail.com>
2019-04-30 21:28:51 +02:00
Michael Crosby
8722ec03c3
Merge pull request #3213 from jcordasc/small-fixes
Correct import path and PusherFunc signature
2019-04-16 15:32:51 -04:00
Jared Cordasco
a7af3c13a5 Correct import path in services/server package
Signed-off-by: Jared Cordasco <jcordasc@coglib.com>
2019-04-11 23:07:46 -05:00
Michael Crosby
97d247cd0f Rename hrpc to tcpServer
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2019-04-11 15:06:06 -04:00
Derek McGowan
5f4c977ba0
Merge pull request #3162 from crosbymichael/tcpservice
Add tcp service for grpc listeners
2019-04-11 11:55:00 -07:00
Derek McGowan
acca107732
Merge pull request #3204 from crosbymichael/fix-forward
Fix API forward events for shims
2019-04-11 11:46:46 -07:00
Phil Estes
5703f415c6
Merge pull request #3197 from Random-Liu/add-required-plugins
Add support for required plugins.
2019-04-11 12:02:16 +02:00
Lantao Liu
4b3b99ea11 Add support for required plugins.
Signed-off-by: Lantao Liu <lantaol@google.com>
2019-04-10 11:31:59 -07:00
Michael Crosby
4ba756edda Fix API forward events for shims
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2019-04-10 13:34:34 -04:00
Michael Crosby
b911ae3428 Add tcp service for grpc listeners
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2019-04-10 11:12:50 -04:00
Michael Crosby
a6f587e4c4 Use ttrpc to publish runtime v2 events
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2019-04-09 14:38:50 -04:00
Michael Crosby
a8a805cad3 Add ttrpc server to containerd
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2019-04-09 13:08:23 -04:00
Stefan Berger
baf3403439 Extend Applier's Apply() method with an optional options parameter
Extend the Applier interface's Apply method with an optional
options parameter.

For the container image encryption we intend to use the options
parameter to pass image decryption parameters ('dcparameters'),
which are primarily (privatte) keys, in form of a JSON document
under the map key '_dcparameters', and pass them to the Applier's
Apply() method. This helps us to access decryption keys and start
the pipeline with the layer decryption before the layer data are
unzipped and untarred.

Signed-off-by: Stefan Berger <stefanb@linux.ibm.com>
Signed-off-by: Harshal Patil <harshal.patil@in.ibm.com>
2019-04-02 18:19:48 -04:00
Michael Crosby
559dfa59a1 Add configurable plugin path
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2019-04-01 16:54:27 -04:00
Michael Crosby
9b882c44f8
Merge pull request #3000 from stefanberger/descriptor_annotations.pr
Add missing annotations map to Descriptor for gRPC transfer
2019-03-22 14:05:44 -04:00
John Howard
6034c1950a Windows:Create root/state with ACL
Signed-off-by: John Howard <jhoward@microsoft.com>
2019-03-21 18:47:34 -07:00
Akihiro Suda
f3ff95ab72 decouple api.ContentServer implementation package from bbolt dependency
For moby/buildkit#886

Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2019-03-19 16:09:45 +09:00
Stefan Berger
0b711d616a Copy annotations around where necessary
Make sure that the newly added annotations are copied around appropriately.

Signed-off-by: Stefan Berger <stefanb@linux.ibm.com>
2019-03-06 12:26:23 -05:00
Michael Crosby
84a24711e8 Add runc.v2 multi-shim
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2019-02-21 11:09:46 -05:00
Wei Fu
132ee9b826 fix: linter issue
megacheck, gosimple and unused has been deprecated and subsumed by
staticcheck. And staticcheck also has been upgraded. we need to update
code for the linter issue.

close: #2945

Signed-off-by: Wei Fu <fuweid89@gmail.com>
2019-01-23 22:54:51 +08:00
Derek McGowan
8706a355dd
Merge pull request #2889 from linxiulei/isolated_content
metadata: define content sharing policy
2019-01-14 13:15:38 -08:00
Eric Lin
4247f2684d metadata: define content sharing policy
This changeset modifies the metadata store to allow one to set a
"content sharing policy" that defines how blobs are shared between
namespaces in the content store.

The default mode "shared" will make blobs available in all namespaces
once it is pulled into any namespace.  The blob will be pulled into
the namespace if a writer is opened with the "Expected" digest that
is already present in the backend.

The alternative mode, "isolated" requires that clients prove they have
access to the content by providing all of the content to the ingest
before the blob is added to the namespace.

Both modes share backing data, while "shared" will reduce total
bandwidth across namespaces, at the cost of allowing access to any
blob just by knowing its digest.

Note: Most functional codes and changelog of this commit originate from
Stephen J Day <stephen.day@docker.com>, see
40455aade8

Fixes #1713 Fixes #2865

Signed-off-by: Eric Lin <linxiulei@gmail.com>
2018-12-21 15:02:21 +08:00
Phil Estes
06e04bc5a9
Merge pull request #2830 from Ace-Tang/support_cr_without_image
cr: support checkpoint/restore without image
2018-12-20 13:24:37 -05:00
zhangyue
6ccb0d0629 fix: remove dead code
Signed-off-by: zhangyue <zy675793960@yeah.net>
2018-12-13 09:55:08 +08:00
Ace-Tang
6593399e9f cr: support checkpoint/restore without image
support checkpoint without committing a checkpoint dir into a
checkpoint image and restore without untar image into checkpoint
directory. support for both v1 and v2 runtime

Signed-off-by: Ace-Tang <aceapril@126.com>
2018-11-29 10:19:39 +08:00
John Howard
40d898a820 Windows: Allow running as a service
Signed-off-by: John Howard <jhoward@microsoft.com>

Allows containerd.exe to run as a Windows service. eg

Register: `.\containerd.exe --register-service`

Start: `net start containerd`
...
Stop: `net stop containerd`

Unregister: `.\containerd.exe --unregister-service`

When running as a service, logs will go to the Windows application
event log.
2018-11-27 17:28:04 -08:00
Phil Estes
f0b081bb44
Merge pull request #2734 from Ace-Tang/start_early_error
check task existence before new task
2018-11-07 10:02:32 -05:00
Justin Terry (VM)
8329a491b9 Build Windows V1 and V2 runtimes always
Signed-off-by: Justin Terry (VM) <juterry@microsoft.com>
2018-11-02 11:59:03 -07:00
Ace-Tang
124194b98b check task existence before new task
make user know correct error when new task.

Signed-off-by: Ace-Tang <aceapril@126.com>
2018-11-03 02:55:12 +08:00
Michael Crosby
35a32a8778
Merge pull request #2747 from fuweid/bugfix_wrap_error_in_closeIO
bugfix: CloseIO should return correct status code
2018-10-29 11:25:37 -04:00
Wei Fu
00ba92364c bugfix: CloseIO should return correct status code
Signed-off-by: Wei Fu <fuweid89@gmail.com>
2018-10-28 12:08:59 +08:00
Michael Crosby
16aaf6c065 Change unsupported snapshot warnings to INFO
Since there is no real action the user can do, these can safely be
informative that the underlying filesystem does not support a snapshot
plugin at boot.

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2018-10-26 12:44:14 -04:00
Wei Fu
06616dab00 enhance: split config from server package
The github.com/containerd/containerd/services/server has a lot of
dependencies, like content, snapshots services implementation and
docker-metrics.

For the client side, it uses the config struct from server package
to start up the containerd in background. It will import a lot of
useless packages which might be conflict with existing vendor's package.

It makes integration easier with single config package.

Signed-off-by: Wei Fu <fuweid89@gmail.com>
2018-10-17 16:34:39 +08:00
Derek McGowan
0f756495a9
Fix writer deadlock in local store
The local store could end up in a state where the writer is
closed but the reference is locked after a commit on an
existing object.
Cleans up Commit logic to always close the writer even after
an error occurs, guaranteeing the reference is unlocked after commit.
Adds a test to the content test suite to verify this behavior.
Updates the content store interface definitions to clarify the behavior.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-10-15 16:33:45 -07:00
Phil Estes
05984a966d
Merge pull request #2642 from dmcgowan/fix-commit-already-exists
Fix content store bug when already exists
2018-09-14 09:09:22 -04:00
Derek McGowan
6875d3df3a
Always check exists on commit error
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-09-14 01:26:03 -07:00
John Howard
2586f3fbb9 boltdb/bolt --> go.etcd.io/bbolt
Signed-off-by: John Howard <jhoward@microsoft.com>
2018-09-12 15:23:57 -07:00
Justin Terry (VM)
0110b3c0bc Introduce the Windows lcow diff/snaphotter
Implements the Windows lcow differ/snapshotter responsible for managing
the creation and lifetime of lcow containers on Windows.

Signed-off-by: Justin Terry (VM) <juterry@microsoft.com>
2018-08-28 13:51:04 -07:00
Michael Crosby
e5115c3919 Export content service New
Closes #2441

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2018-08-21 15:38:51 -04:00
Derek McGowan
dd97a11b6f
Merge pull request #2523 from jterry75/windows_v2_tasks_service
Refactors the TasksService requires per platform
2018-08-06 10:33:46 -07:00
Michael Crosby
5a47c5ec1d Add lib support as an option
Some images like `criu` will have extra libs that it requires.  This
adds lib support via LD_LIBRARY_PATH and InstallOpts

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2018-08-02 17:11:29 -04:00
Evan Hazlett
1537f31381 Add install support for binary images
This adds a way for users to programatically install containerd binary
dependencies.

With runtime v2 and new shim's being built, it will be a challenge to
get those onto machines.  Users would have to find the link, download,
place it in their path, yada yada yada.

With this functionality of a managed `/opt` directory, containerd can
use existing image and distribution infra. to get binarys, shims, etc
onto the system.

Configuration:

*default:* `/opt/containerd`

*containerd config:*
```toml
[plugins.opt]
	path = "/opt/mypath"

```

Usage:

*code:*

```go
image, err := client.Pull(ctx, "docker.io/crosbymichael/runc:latest")
client.Install(ctx, image)
```

*ctr:*

```bash
ctr content fetch docker.io/crosbymichael/runc:latest
ctr install docker.io/crosbymichael/runc:latest
```

You can manage versions and see what is running via standard image
commands.

Images:

These images MUST be small and only contain binaries.

```Dockerfile
FROM scratch
Add runc /bin/runc
```

Containerd will only extract files in `/bin` of the image.

Later on, we can add support for `/lib`.

The code adds a service to manage an `/opt/containerd` directory and
provide that path to callers via the introspection service.

How to Test:

Delete runc from your system.

```bash
> sudo ctr run --rm  docker.io/library/redis:alpine redis
ctr: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/containerd/io.containerd.runtime.v1.linux/default/redis/log.json: no such file or directory): exec: "runc": executable file not found in $PATH: unknown

> sudo ctr content fetch docker.io/crosbymichael/runc:latest
> sudo ctr  install docker.io/crosbymichael/runc:latest

> sudo ctr run --rm  docker.io/library/redis:alpine redis
1:C 01 Aug 15:59:52.864 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 01 Aug 15:59:52.864 # Redis version=4.0.10, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 01 Aug 15:59:52.864 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
1:M 01 Aug 15:59:52.866 # You requested maxclients of 10000 requiring at least 10032 max file descriptors.
1:M 01 Aug 15:59:52.866 # Server can't set maximum open files to 10032 because of OS error: Operation not permitted.
1:M 01 Aug 15:59:52.866 # Current maximum open files is 1024. maxclients has been reduced to 992 to compensate for low ulimit. If you need higher maxclients increase 'ulimit -n'.
1:M 01 Aug 15:59:52.870 * Running mode=standalone, port=6379.
1:M 01 Aug 15:59:52.870 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:M 01 Aug 15:59:52.870 # Server initialized
1:M 01 Aug 15:59:52.870 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
1:M 01 Aug 15:59:52.870 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
1:M 01 Aug 15:59:52.870 * Ready to accept connections
^C1:signal-handler (1533139193) Received SIGINT scheduling shutdown...
1:M 01 Aug 15:59:53.472 # User requested shutdown...
1:M 01 Aug 15:59:53.472 * Saving the final RDB snapshot before exiting.
1:M 01 Aug 15:59:53.484 * DB saved on disk
1:M 01 Aug 15:59:53.484 # Redis is now ready to exit, bye bye...
```

Signed-off-by: Evan Hazlett <ejhazlett@gmail.com>
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2018-08-02 17:11:29 -04:00
Justin Terry (VM)
9936370fef Refactors the TasksService requires per platform
Removes the start dependency on V1 runtimes in the TasksService for:
// +build windows_v2. For unix and windows (v1) this code remains to load all
v1 runtimes as expected.

Signed-off-by: Justin Terry (VM) <juterry@microsoft.com>
2018-08-02 11:20:35 -07:00
Phil Estes
4249f44d81
Merge pull request #2493 from dmcgowan/sync-lease-removal
Add sync option to lease removal
2018-07-31 11:31:55 -04:00
Michael Crosby
400f16fc46 Add containers streaming API
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2018-07-26 13:55:20 -04:00
Derek McGowan
b760cee65a
Update lease service errors
Ensure delete returns a typed error on not found

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-07-25 14:56:35 -07:00
Michael Crosby
0d52c71c80
Merge pull request #2474 from dmcgowan/lease-expiration
Improved lease management
2018-07-20 16:54:17 -04:00
Derek McGowan
94e132fd07
Add sync option on lease removal
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-07-19 14:37:55 -07:00
Derek McGowan
94cfce62ba
Merge pull request #2434 from crosbymichael/shimv2
Runtime v2 (shim API)
2018-07-18 13:14:10 -07:00
Derek McGowan
8cf3fad8d4
Add leases manager interface
Add leases manager to the leases package and use the
interface on the client and service.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-07-18 10:43:37 -07:00
Akihiro Suda
02579c8c3f
Merge pull request #2415 from dmcgowan/proxy-plugins
Add support for proxy plugins
2018-07-18 11:17:24 +09:00
Michael Crosby
7e49c601a8 Add shim start for shim creation
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2018-07-17 10:21:29 -04:00
Michael Crosby
da1b5470cd Runtime v2
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2018-07-17 10:21:29 -04:00
Dave Henderson
9a97ab34ce
Switching from crypto/rand to math/rand to avoid blocking
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
2018-07-11 16:24:51 -04:00
Derek McGowan
7049671465
Add support for proxy plugins in configuration
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-06-21 18:57:42 -07:00
Vincent Demeester
832b05ae67
Update tests to use gotest.tools angel
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
2018-06-08 21:02:01 +02:00
Derek McGowan
993d4b8fc5
Ignore zero max message size in grpc config
Fixes upgrade issue where existing configs cause
the max message size to get set to 0.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-06-01 16:21:53 -07:00
Akihiro Suda
d88de4a34f content: change Writer/ReaderAt to take OCI
This change allows implementations to resolve the location of the actual data
using OCI descriptor fields such as MediaType.

No OCI descriptor field is written to the store.

No change on gRPC API.

Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2018-06-01 11:51:43 +09:00
Michael Crosby
8e97da0958 Move server to services pkg
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2018-05-22 17:15:58 -04:00
Michael Crosby
ceae112f7d Rename Runtime to PlatformRuntime
This renames the runtime interface to PlatformRuntime to denote the
layer at which the runtime is being abstracted.  This should be used to
abstract different platforms that vary greatly and do not have full
compat with OCI based binary runtimes.

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2018-05-16 09:36:05 -04:00
Tom Godkin
fc8bce59b9 Use user-specific temp directory if set
This allows non-privileged users to use containerd. This is part of a
larger track of work integrating containerd into Cloudfoundry's garden
with support for rootless.

[#156343575]

Signed-off-by: Claudia Beresford <cberesford@pivotal.io>
2018-05-04 10:27:58 +01:00
Kir Kolyshkin
bbe14f0a2e Switch from x/net/context to context
Since Go 1.7, context is a standard package, superceding the
"x/net/context". Since Go 1.9, the latter only provides a few type
aliases from the former. Therefore, it makes sense to switch to the
standard package.

This commit was generated by the following script (with a couple of
minor fixups to remove extra changes done by goimports):

	#!/bin/bash

	if [ $# -ge 1 ]; then
		FILES=$*
	else
		FILES=$(git ls-files \*.go | grep -vF ".pb.go" | grep -v
	^vendor/)
	fi

	for f in $FILES; do
		printf .
		sed -i -e 's|"golang.org/x/net/context"$|"context"|' $f
		goimports -w $f
		awk '	/^$/ {e=1; next;}
			/[[:space:]]"context"$/ {e=0;}
			{if (e) {print ""; e=0}; print;}' < $f > $f.new && \
				mv $f.new $f
		goimports -w $f
	done
	echo

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2018-04-24 14:33:34 -07:00
Kir Kolyshkin
9d0d4b806c context pkg: untangle
Since Go 1.7, "context" is a standard package, superceding the
"x/net/context". Since Go 1.9, the latter only provides type aliases
from the former. Therefore, it makes sense to switch to the standard
package, and the change is not disruptive in any sense.

This commit deals with a few cases where both packages happened to be
imported by the same source file. A choice between "context" and
"gocontext" was made for each file in order to minimize the patch.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2018-04-24 14:33:34 -07:00
Michael Wan
7fd6d5e2be fix words misspell
Signed-off-by: Michael Wan <zirenwan@gmail.com>
2018-04-23 00:09:42 -04:00
Stephen J Day
caac3bca3e
protobuf: remove generated google/rpc files
We move from having our own generated version of the googleapis files to
an upstream version that is present in gogo. As part of this, we update
the protobuf package to 1.0 and make some corrections for slight
differences in the generated code.

The impact of this change is very low.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2018-04-05 14:36:32 -07:00
Akihiro Suda
7b323b1402 services/content: fix reading a blob which is smaller than the read buffer.
The newly added test fails without this fix in services/content/service.go:

    $ go test -c . && sudo ./containerd.test -test.v -test.root -test.run TestContentClient
    ...
        --- FAIL: TestContentClient/SmallBlob (0.02s)
            provideringester.go:62: rpc error: code = OutOfRange desc = read
    past object length 6 bytes
            helpers.go:67: drwx------       4096
    /tmp/content-suite-ContentClient-286788688
    FAIL

Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2018-03-23 12:51:28 +09:00
Stephen Day
b3b95c0a2a
Merge pull request #2154 from dmcgowan/shared-content-ingests
content: shared content across namespaces
2018-03-12 16:11:32 -07:00
Lantao Liu
1128b3d664 Add service plugin and support in process integration.
Signed-off-by: Lantao Liu <lantaol@google.com>
2018-03-12 18:03:50 +00:00
Derek McGowan
a1a67899f8
Shared content across namespaces
Update content ingests to use content from another namespace.
Ingests must be committed to make content available and the
client will see the sharing as an ingest which has already
been fully written to, but not completed.

Updated the database version to change the ingest record in
the database from a link key to an object with a link and
expected value. This expected value is used to indicate that
the content already exists and an underlying writer may
not yet exist.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-02-22 14:45:10 -08:00
Kunal Kushwaha
b12c3215a0 Licence header added
Signed-off-by: Kunal Kushwaha <kushwaha_kunal_v7@lab.ntt.co.jp>
2018-02-19 10:32:26 +09:00
Derek McGowan
b763777288
diff: rename differ to comparer
Remove combined interface and split implementations.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-01-26 16:32:09 -08:00
Akihiro Suda
b580441f91
diff: resplit Applier from Differ
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2018-01-25 13:58:34 -08:00
Darren Stahl
a5a9f91832 Implement Windows snapshotter and differ
This implements the Windows snapshotter and diff Apply function.
This allows for Windows layers to be created, and layers to be pulled
from the hub.

Signed-off-by: Darren Stahl <darst@microsoft.com>
2018-01-23 14:40:23 -08:00
Stephen Day
acc6011ac1
Merge pull request #1960 from dmcgowan/images-removal-dirty
metadata: image removal triggers GC
2018-01-17 13:58:39 -08:00
Phil Estes
0b449a4a51
Merge pull request #1982 from stevvooe/deletion-race-handling
services/tasks, linux: ignore shutdown tasks
2018-01-10 16:53:42 -05:00
Stephen J Day
c80ca4f4a2
services/tasks, linux: ignore shutdown tasks
Because tasks may be deleted while listing containers, we need to ignore
errors from state requests that are due to a closed error. All of these
get mapped to ErrNotFound, which can be used to filter the entries.

There may be a better fix that does a better job of keeping track of the
intended state of a backend task. The current condition of assuming that
a closed client is a shutdown task may be too naive.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2018-01-10 11:45:33 -08:00
Daniel Nephin
184bc25629 Add unconvert linter
This linter checks for unnecessary type convertions.

Some convertions are whitelisted because their type is different
on 32bit platforms

Signed-off-by: Daniel Nephin <dnephin@gmail.com>
2018-01-09 17:36:44 -05:00
Derek McGowan
89fa154efd
Update metadata image store to be initialized once
The boltdb image store now manages its own transactions when
one is not provided, but allows the caller to pass in a
transaction through the context. This makes the image store
more similar to the content and snapshot stores. Additionally,
use the reference to the metadata database to mark the content
store as dirty after an image has been deleted. The deletion
of an image means a reference to a piece of content is gone
and therefore garbage collection should be run to check if
any resources can be cleaned up as a result.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-01-05 13:38:45 -08:00
Daniel Nephin
06edd193ef Small refactor of gc/scheduler to remove import of metadata
Replace metadata.GCStats with an interface for exposing elapsed time

Signed-off-by: Daniel Nephin <dnephin@gmail.com>
2018-01-03 14:36:11 -05:00
Derek McGowan
b28d7cdf1b
Update snapshot and content related log messages
Make the tense and casing consistent.
Add useful log messages in image service.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-12-01 14:57:34 -08:00
Jess Valarezo
1dd6f339a2 rename snapshot->snapshots proto pkg and service
Signed-off-by: Jess Valarezo <valarezo.jessica@gmail.com>
2017-11-29 14:55:24 -08:00
Jess Valarezo
9885edfc44 rename snapshot->snapshots pkg
Signed-off-by: Jess Valarezo <valarezo.jessica@gmail.com>
2017-11-29 14:55:02 -08:00
Stephen Day
90a6b79bfc
Merge pull request #1810 from dnephin/add-staticcheck-linter
Fix usage of sync.Pool
2017-11-28 15:46:41 -08:00
Daniel Nephin
dab3ba2aff Replace calls to deprecated grpc functions
Signed-off-by: Daniel Nephin <dnephin@gmail.com>
2017-11-28 15:12:05 -05:00
Daniel Nephin
ee04cfa3f9 Add staticcheck linter
Fix issues with sync.Pool being passed an array and not a pointer.
See https://github.com/dominikh/go-tools/blob/master/cmd/staticcheck/docs/checks/SA6002

Add missing tests for content.Copy

Fix T.Fatal being called in a goroutine

Signed-off-by: Daniel Nephin <dnephin@gmail.com>
2017-11-28 13:05:30 -05:00
Stephen Day
372cdfac3b
Merge pull request #1638 from dmcgowan/gc-policy
gc: add policy plugin
2017-11-27 18:20:10 -08:00
Derek McGowan
bae47820d7
Document defaults
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-11-21 16:12:10 -08:00
Daniel Nephin
f74862a0dd Add structcheck, unused, and varcheck linters.
Warn on unused and dead code

Signed-off-by: Daniel Nephin <dnephin@gmail.com>
2017-11-21 11:14:37 -05:00
Derek McGowan
3f1a61f76a
Add synchronous image delete
Synchronous image delete provides an option image delete to wait
until the next garbage collection deletes after an image is removed
before returning success to the caller.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-11-20 17:08:35 -08:00
Derek McGowan
00596f400e
Add gc policy plugin
Add garbage collection as a background process and policy
configuration for configuring when to run garbage collection.
By default garbage collection will run when deletion occurs
and no more than 20ms out of every second.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-11-20 16:57:39 -08:00
Phil Estes
2edc475818
Merge pull request #1776 from stevvooe/decouple-events-grpc
events: decouple events package from grpc
2017-11-17 11:44:46 -05:00
Michael Crosby
a4968710fd
Merge pull request #1777 from jessvalarezo/migrate-client-code
client: migrate client code to containerd
2017-11-17 09:32:34 -05:00
Stephen J Day
2d05c4aa1a
events: decouple events package from grpc
By defining a concrete, non-protobuf type for the events interface, we
can completely decouple it from the grpc packages that are expensive at
runtime. This does requires some allocation cost for converting between
types, but the saving for the size of the shim are worth it.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-11-16 18:49:46 -08:00
Jess Valarezo
8c5dede74e client: move content store client to containerd package
Signed-off-by: Jess Valarezo <valarezo.jessica@gmail.com>
2017-11-16 18:14:21 -08:00
Jess Valarezo
aefaeeacb0 client: move diff client to containerd package
Signed-off-by: Jess Valarezo <valarezo.jessica@gmail.com>
2017-11-16 18:06:14 -08:00
Jess Valarezo
8d23e5ef9c client: move image store client to containerd package
Signed-off-by: Jess Valarezo <valarezo.jessica@gmail.com>
2017-11-16 17:47:54 -08:00
Jess Valarezo
6cd9962c7f client: move namespaces client to containerd package
Signed-off-by: Jess Valarezo <valarezo.jessica@gmail.com>
2017-11-16 17:47:38 -08:00
Derek McGowan
5c6e789dde
Merge pull request #1769 from stevvooe/split-events-types
api/events: split event types from events service
2017-11-16 16:29:28 -08:00
Jess Valarezo
5b64f7030d client: move snapshot client to containerd package
Signed-off-by: Jess Valarezo <valarezo.jessica@gmail.com>
2017-11-16 15:45:07 -08:00
Stephen J Day
09b5ca1072
api/events: split event types from events service
To avoid importing all of grpc when consuming events, the types of
events have been split in to a separate package. This should allow a
reduction in memory usage in cases where a package is consuming events
but not using the gprc service directly.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-11-16 15:20:46 -08:00
Stephen J Day
3e5e2ecc0a content/local: consistent handling of data and locks
The locks now retry on the backend side to prevent clients from having
to round trip on locks that might be momentarily held. This exposed some
timing errors in the updated_at fields for content ingest, so we've had
to move that to a separate file to export the monotonic go runtime
timestamps.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-11-16 16:13:19 -05:00
Stephen J Day
c5022ad92d
protobuf: use the gogo/types package for empty
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-11-15 19:08:54 -08:00
Akihiro Suda
4feb6f228a snapshot: add Close()
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-11-14 02:00:33 +00:00
Jess Valarezo
807f4d2ec7 expose exec-id on ctr task ps
Signed-off-by: Jess Valarezo <valarezo.jessica@gmail.com>
2017-11-10 11:38:51 -08:00
Stephen Day
27d450a01b
Merge pull request #1690 from dmcgowan/metadata-transactions
gc: resource leases
2017-11-07 16:11:11 -08:00
Derek McGowan
dce27d8c62
Remove client use of gc root label
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-11-07 12:54:22 -08:00
Derek McGowan
e13894bb7a
Add leases api
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-11-07 12:54:22 -08:00
Michael Crosby
a70b95b202 Move events exchange into subpackage
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-11-07 10:51:12 -05:00
Michael Crosby
b6e0c4f321 Fix go lint errors
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-10-25 15:26:44 -04:00
Michael Crosby
5fd0415985 Add comments and fix common lint issues
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-10-20 13:19:14 -04:00
Michael Crosby
9bd1dc78cb Unexport grpc service types
Since these are registered and the interface is what matters, these
Service types do not need to be exported.

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-10-20 13:19:14 -04:00
Derek McGowan
ffb03c4fe2
Add checkpoint and userns gc labels
Prevent checkpoints from getting garbage collected by
adding root labels to unreferenced checkpoint objects.
Mark checkpoints as gc roots.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-10-11 11:26:46 -07:00
Derek McGowan
de7b281856
Update services to call garbage collection
Call garbage collection on container and image deletion.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-10-11 10:42:47 -07:00
Derek McGowan
d9db1d112d
Refactor differ into separate package
Add differ options and package with interface.
Update optional values on diff interface to use options.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-10-11 10:02:29 -07:00
Stephen J Day
0e72ce5c5b
services/introspection: implement plugin reporting
With this change, we integrate all the plugin changes into the
introspection service.

All plugins can be listed with the following command:

```console
$ ctr plugins
TYPE                            ID             PLATFORM       STATUS
io.containerd.content.v1        content        -              ok
io.containerd.metadata.v1       bolt           -              ok
io.containerd.differ.v1         walking        linux/amd64    ok
io.containerd.grpc.v1           containers     -              ok
io.containerd.grpc.v1           content        -              ok
io.containerd.grpc.v1           diff           -              ok
io.containerd.grpc.v1           events         -              ok
io.containerd.grpc.v1           healthcheck    -              ok
io.containerd.grpc.v1           images         -              ok
io.containerd.grpc.v1           namespaces     -              ok
io.containerd.snapshotter.v1    btrfs          linux/amd64    error
io.containerd.snapshotter.v1    overlayfs      linux/amd64    ok
io.containerd.grpc.v1           snapshots      -              ok
io.containerd.monitor.v1        cgroups        linux/amd64    ok
io.containerd.runtime.v1        linux          linux/amd64    ok
io.containerd.grpc.v1           tasks          -              ok
io.containerd.grpc.v1           version        -              ok
```

There are few things to note about this output. The first is that it is
printed in the order in which plugins are initialized. This useful for
debugging plugin initialization problems. Also note that even though the
introspection GPRC api is a itself a plugin, it is not listed. This is
because the plugin takes a snapshot of the initialization state at the
end of the plugin init process. This allows us to see errors from each
plugin, as they happen. If it is required to introspect the existence of
the introspection service, we can make modifications to include it in
the future.

The last thing to note is that the btrfs plugin is in an error state.
This is a common state for containerd because even though we load the
plugin, most installations aren't on top of btrfs and the plugin cannot
be used. We can actually view this error using the detailed view with a
filter:

```console
$ ctr plugins --detailed id==btrfs
Type:          io.containerd.snapshotter.v1
ID:            btrfs
Platforms:     linux/amd64
Exports:
               root      /var/lib/containerd/io.containerd.snapshotter.v1.btrfs
Error:
               Code:        Unknown
               Message:     path /var/lib/containerd/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter
```

Along with several other values, this is a valuable tool for evaluating the
state of components in containerd.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-10-10 17:29:24 -07:00
Stephen J Day
8508e8252b
plugin: refactor plugin system to support initialization reporting
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-10-10 16:40:47 -07:00
Jess
061c719209 ListPids returns process ID and other info
Signed-off-by: Jess <jessica.valarezo@docker.com>
2017-10-10 22:57:15 +00:00
Phil Estes
987fcd1201 Merge pull request #1598 from Random-Liu/fix-load-task
Fix task load.
2017-10-06 16:38:40 -04:00
Lantao Liu
28ca8f05d3 Fix task load.
Signed-off-by: Lantao Liu <lantaol@google.com>
2017-10-05 21:03:24 +00:00
Derek McGowan
8d892a651b
Update metadata plugin initialization
Updates metadata plugin to require content and
snapshotter plugins be loaded and initializes with
those plugins, keeping the metadata database structure
static after initialization. Service plugins now only
require metadata plugin access snapshotter or content
stores through metadata, which was already required
behavior of the services.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-10-04 11:25:46 -07:00
Derek McGowan
56c1b79a4c
Create metadata db object
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-10-04 11:16:22 -07:00
Michael Crosby
451421b615 Comment more packages to pass go lint
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-10-02 13:54:56 -04:00
Stephen Day
df896c92e7 Merge pull request #1528 from jessvalarezo/labels-validate
Labels are consistently validated across services.
2017-09-22 10:33:17 -07:00
Lantao Liu
54521126aa Fix container CreatedAt and UpdatedAt.
Signed-off-by: Lantao Liu <lantaol@google.com>
2017-09-22 05:57:02 +00:00
Jess Valarezo
18c4322bb3 Labels are consistently validated across services
* The combined size of a key/value pair cannot exceed 4096 bytes

Signed-off-by: Jess Valarezo <valarezo.jessica@gmail.com>
2017-09-21 15:11:46 -07:00
Michael Crosby
d67763d922 Add wait API endpoint for waiting on process exit
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-09-21 15:03:58 -04:00
Michael Crosby
d22160c28e Vendor typeurl package
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-09-19 09:43:55 -04:00
Kenfe-Mickaël Laventure
e517952bc7 Merge pull request #1378 from cpuguy83/container_store_extra_data
Add `Container` field to store client-defined data
2017-09-18 07:32:07 -07:00
Michael Crosby
72f380af45 Merge pull request #1517 from mlaventure/remove-toml-omitempty
Remove omitempty from toml tags
2017-09-15 11:30:01 -04:00
Kenfe-Mickael Laventure
f2d1459929
Remove omitempty from toml tags
The encoder only support changing the name of the key.

Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
2017-09-14 20:49:22 -07:00
Michael Crosby
14c36cca60 Set ctx ns before publish
Fixes #1497

This sets the namespace on the context when deleting a namespace so that
the publish event does not fail.  Use the same namespace as you are
deleting for the context.

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-09-14 13:54:57 -04:00
Derek McGowan
8785f3f002 Merge pull request #1403 from stevvooe/platform-selectors
platforms: define selectors for platforms
2017-09-13 11:50:11 -07:00
Stephen J Day
94f6be5f10
platforms: implement matcher support
Matching support is now implemented in the platforms package. The
`Parse` function now returns a matcher object that can be used to
match OCI platform specifications. We define this as an interface to
allow the creation of helpers oriented around platform selection.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-09-08 19:22:29 -07:00
Brian Goff
3552ce5688 Add field to Container for client-defined data
This field allows a client to store specialized information in the
container metadata rather than having to store this itself and keep
the data in sync with containerd.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2017-09-07 16:54:47 -04:00
Michael Crosby
951c129bf1 Handle locking and errors for process state
ref: #1464

This tries to solve issues with races around process state.  First it
adds the process mutex around the state call so that any state changes,
deletions, etc will be handled in order.

Second, for IsNoExist errors from the runtime, return a stopped state if
a process has been removed from the underlying OCI runtime but not from
the shim yet.  This shouldn't happen with the lock from above but its
hare to verify this issue.

Third, handle shim disconnections and return an ErrNotFound.

Forth, don't abort returning all tasks if one task is unable to return
its state.

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-09-07 16:22:00 -04:00
Akihiro Suda
e30e0c8b75
api: RootFS -> SnapshotKey
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-09-06 15:22:19 -07:00
Kenfe-Mickaël Laventure
e66d1a7961 Merge pull request #1468 from crosbymichael/stats
Add Metrics endpoint to tasks
2017-09-06 14:59:55 -07:00
Kenfe-Mickaël Laventure
e1eeb0e0a2 Merge pull request #1475 from dmcgowan/content-commit-context
Add context to content commit
2017-09-06 11:04:31 -07:00
Derek McGowan
9613acb2ed
Add context to content commit
Content commit is updated to take in a context, allowing
content to be committed within the same context the writer
was in. This is useful when commit may be able to use more
context to complete the action rather than creating its own.
An example of this being useful is for the metadata implementation
of content, having a context allows tests to fully create
content in one database transaction by making use of the context.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-09-06 10:19:12 -07:00
Michael Crosby
4c5ed9c068 Move metrics requests to services
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-09-05 17:41:30 -04:00
Michael Crosby
8510512e7e Add test for client metrics
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-09-05 17:26:26 -04:00
Michael Crosby
697dcdd407 Refactor task service metrics
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-09-05 17:26:26 -04:00
Michael Crosby
f5d81a631e Return grpc errs from task service
Closes #1201

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-09-05 16:10:19 -04:00
Stephen J Day
5cc108605f
service/containers: correctly plumb fieldpaths
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-09-01 13:17:31 -07:00
Kenfe-Mickaël Laventure
0daa593b3a Merge pull request #1393 from dmcgowan/multiple-differs
Add support for multiple differs
2017-08-23 14:45:17 -07:00
Michael Crosby
873a34649a Merge pull request #1366 from stevvooe/containers-metadata-tests
metadata: ensure correct updates on Container
2017-08-21 16:08:06 -04:00
Kenfe-Mickael Laventure
8a1b03e525
Add ExitedAt to process proto definition
Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
2017-08-21 08:18:02 -07:00
Stephen J Day
677257f032
service/snapshotter: move default to client
In order to enforce strict handling of snapshotter values on the
container object, the defaults have been moved to the client side. This
ensures that we correctly qualify the snapshotter under use when from
the container at the time it was created, rather than possibly losing
the metadata on a change of default.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-08-18 14:43:12 -07:00
Derek McGowan
2fa366d6af
Add support for multiple differs
Updates the differ service to support calling and configuring
multiple differs. The differs are configured as an ordered list
of differs which will each be attempting until a supported differ
is called.

Additionally a not supported error type was added to allow differs
to be selective of whether the differ arguments are supported by
the differ. This error type corresponds to the GRPC unimplemented error.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-08-18 11:19:08 -07:00
Derek McGowan
dee8dc2cda
Add support for content labels on commit
Add commit options which allow for setting labels on commit.
Prevents potential race between garbage collector reading labels
after commit and labels getting set.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-08-11 14:15:20 -07:00
Derek McGowan
c4387a159e
Add content test suite run to client
Fix bugs in content deletion and upload status

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-08-11 14:08:34 -07:00
Stephen Day
e6ba48f25c Merge pull request #1257 from stevvooe/content-changes
content: remove Provider.Reader
2017-08-09 17:12:44 -07:00
Stephen J Day
8be340e37b
content: remove Provider.Reader
After some analysis, it was found that Content.Reader was generally
redudant to an io.ReaderAt. This change removes `Content.Reader` in
favor of a `Content.ReaderAt`. In general, `ReaderAt` can perform better
over interfaces with indeterminant latency because it avoids remote
state for reads. Where a reader is required, a helper is provided to
convert it into an `io.SectionReader`.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-08-09 14:32:28 -07:00
Michael Crosby
29a4dd7f46 Merge pull request #1276 from dmcgowan/snapshot-labels
Snapshot labels
2017-08-08 21:59:28 -04:00
Michael Crosby
2fb2168bae Merge pull request #1224 from epilatow/solaris.cc.20170719.0
Add solaris build support
2017-08-07 16:15:07 -04:00
Edward Pilatowicz
56c1f5c184 Add solaris build support
Signed-off-by: Edward Pilatowicz <edward.pilatowicz@oracle.com>
2017-08-07 12:40:21 -07:00
Michael Crosby
9f13b414b9 Return exit status from Wait of stopped process
This changes Wait() from returning an error whenever you call wait on a
stopped process/task to returning the exit status from the process.

This also adds the exit status to the Status() call on a process/task so
that a user can Wait(), check status, then cancel the wait to avoid
races in event handling.

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-08-03 17:22:33 -04:00
Derek McGowan
efca0c5347
Update GRPC api to support labels and snapshot updates
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-08-02 16:28:41 -07:00
Michael Crosby
504033e373 Add Get of task and process state
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-08-02 13:50:08 -04:00
Michael Crosby
a2a3451925 Implement Exec + Start for tasks service
This splits up the exec creation and start in the tasks service

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-08-02 13:50:08 -04:00
Stephen J Day
af2d7f0e55
events: initial support for filters
This change further plumbs the components required for implementing
event filters. Specifically, we now have the ability to filter on the
`topic` and `namespace`.

In the course of implementing this functionality, it was found that
there were mismatches in the events API that created extra serialization
round trips. A modification to `typeurl.MarshalAny` and a clear
separation between publishing and forwarding allow us to avoid these
serialization issues.

Unfortunately, this has required a few tweaks to the GRPC API, so this
is a breaking change. `Publish` and `Forward` have been clearly separated in
the GRPC API. `Publish` honors the contextual namespace and performs
timestamping while `Forward` simply validates and forwards. The behavior
of `Subscribe` is to propagate events for all namespaces unless
specifically filtered (and hence the relation to this particular change.

The following is an example of using filters to monitor the task events
generated while running the [bucketbench tool](https://github.com/estesp/bucketbench):

```
$ ctr events 'topic~=/tasks/.+,namespace==bb'
...
2017-07-28 22:19:51.78944874 +0000 UTC   bb        /tasks/start   {"container_id":"bb-ctr-6-8","pid":25889}
2017-07-28 22:19:51.791893688 +0000 UTC   bb        /tasks/start   {"container_id":"bb-ctr-4-8","pid":25882}
2017-07-28 22:19:51.792608389 +0000 UTC   bb        /tasks/start   {"container_id":"bb-ctr-2-9","pid":25860}
2017-07-28 22:19:51.793035217 +0000 UTC   bb        /tasks/start   {"container_id":"bb-ctr-5-6","pid":25869}
2017-07-28 22:19:51.802659622 +0000 UTC   bb        /tasks/start   {"container_id":"bb-ctr-0-7","pid":25877}
2017-07-28 22:19:51.805192898 +0000 UTC   bb        /tasks/start   {"container_id":"bb-ctr-3-6","pid":25856}
2017-07-28 22:19:51.832374931 +0000 UTC   bb        /tasks/exit   {"container_id":"bb-ctr-8-6","id":"bb-ctr-8-6","pid":25864,"exited_at":"2017-07-28T22:19:51.832013043Z"}
2017-07-28 22:19:51.84001249 +0000 UTC   bb        /tasks/exit   {"container_id":"bb-ctr-2-9","id":"bb-ctr-2-9","pid":25860,"exited_at":"2017-07-28T22:19:51.839717714Z"}
2017-07-28 22:19:51.840272635 +0000 UTC   bb        /tasks/exit   {"container_id":"bb-ctr-7-6","id":"bb-ctr-7-6","pid":25855,"exited_at":"2017-07-28T22:19:51.839796335Z"}
...
```

In addition to the events changes, we now display the namespace origin
of the event in the cli tool.

This will be followed by a PR to add individual field filtering for the
events API for each event type.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-07-31 12:53:18 -07:00
Michael Crosby
b84817a29c Merge pull request #1246 from stevvooe/events-refactor
events: refactor event distribution
2017-07-26 08:31:12 -04:00