Commit Graph

192 Commits

Author SHA1 Message Date
Michael Crosby
504033e373 Add Get of task and process state
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-08-02 13:50:08 -04:00
Michael Crosby
a2a3451925 Implement Exec + Start for tasks service
This splits up the exec creation and start in the tasks service

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-08-02 13:50:08 -04:00
Stephen J Day
af2d7f0e55
events: initial support for filters
This change further plumbs the components required for implementing
event filters. Specifically, we now have the ability to filter on the
`topic` and `namespace`.

In the course of implementing this functionality, it was found that
there were mismatches in the events API that created extra serialization
round trips. A modification to `typeurl.MarshalAny` and a clear
separation between publishing and forwarding allow us to avoid these
serialization issues.

Unfortunately, this has required a few tweaks to the GRPC API, so this
is a breaking change. `Publish` and `Forward` have been clearly separated in
the GRPC API. `Publish` honors the contextual namespace and performs
timestamping while `Forward` simply validates and forwards. The behavior
of `Subscribe` is to propagate events for all namespaces unless
specifically filtered (and hence the relation to this particular change.

The following is an example of using filters to monitor the task events
generated while running the [bucketbench tool](https://github.com/estesp/bucketbench):

```
$ ctr events 'topic~=/tasks/.+,namespace==bb'
...
2017-07-28 22:19:51.78944874 +0000 UTC   bb        /tasks/start   {"container_id":"bb-ctr-6-8","pid":25889}
2017-07-28 22:19:51.791893688 +0000 UTC   bb        /tasks/start   {"container_id":"bb-ctr-4-8","pid":25882}
2017-07-28 22:19:51.792608389 +0000 UTC   bb        /tasks/start   {"container_id":"bb-ctr-2-9","pid":25860}
2017-07-28 22:19:51.793035217 +0000 UTC   bb        /tasks/start   {"container_id":"bb-ctr-5-6","pid":25869}
2017-07-28 22:19:51.802659622 +0000 UTC   bb        /tasks/start   {"container_id":"bb-ctr-0-7","pid":25877}
2017-07-28 22:19:51.805192898 +0000 UTC   bb        /tasks/start   {"container_id":"bb-ctr-3-6","pid":25856}
2017-07-28 22:19:51.832374931 +0000 UTC   bb        /tasks/exit   {"container_id":"bb-ctr-8-6","id":"bb-ctr-8-6","pid":25864,"exited_at":"2017-07-28T22:19:51.832013043Z"}
2017-07-28 22:19:51.84001249 +0000 UTC   bb        /tasks/exit   {"container_id":"bb-ctr-2-9","id":"bb-ctr-2-9","pid":25860,"exited_at":"2017-07-28T22:19:51.839717714Z"}
2017-07-28 22:19:51.840272635 +0000 UTC   bb        /tasks/exit   {"container_id":"bb-ctr-7-6","id":"bb-ctr-7-6","pid":25855,"exited_at":"2017-07-28T22:19:51.839796335Z"}
...
```

In addition to the events changes, we now display the namespace origin
of the event in the cli tool.

This will be followed by a PR to add individual field filtering for the
events API for each event type.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-07-31 12:53:18 -07:00
Michael Crosby
b84817a29c Merge pull request #1246 from stevvooe/events-refactor
events: refactor event distribution
2017-07-26 08:31:12 -04:00
Stephen J Day
a615a6fe5d
events: refactor event distribution
In the course of setting out to add filters and address some cleanup, it
was found that we had a few problems in the events subsystem that needed
addressing before moving forward.

The biggest change was to move to the more standard terminology of
publish and subscribe. We make this terminology change across the Go
interface and the GRPC API, making the behavior more familier. The
previous system was very context-oriented, which is no longer required.

With this, we've removed a large amount of dead and unneeded code. Event
transactions, context storage and the concept of `Poster` is gone. This
has been replaced in most places with a `Publisher`, which matches the
actual usage throughout the codebase, removing the need for helpers.

There are still some questions around the way events are handled in the
shim. Right now, we've preserved some of the existing bugs which may
require more extensive changes to resolve correctly.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-07-25 15:08:09 -07:00
Stephen J Day
863784f991
snapshot: replace "readonly" with View snapshot type
What started out as a simple PR to remove the "Readonly" column became an
adventure to add a proper type for a "View" snapshot. The short story here is
that we now get the following output:

```
$ sudo ctr snapshot ls
ID 									 PARENT 								 KIND
sha256:08c2295a7fa5c220b0f60c994362d290429ad92f6e0235509db91582809442f3 								  	 Committed
testing4								 sha256:08c2295a7fa5c220b0f60c994362d290429ad92f6e0235509db91582809442f3 Active
```

In pursuing this output, it was found that the idea of having "readonly" as an
attribute on all snapshots was redundant. For committed, they are always
readonly, as they are not accessible without an active snapshot. For active
snapshots that were views, we'd have to check the type before interpreting
"readonly". With this PR, this is baked fully into the kind of snapshot. When
`Snapshotter.View` is  called, the kind of snapshot is `KindView`, and the
storage system reflects this end to end.

Unfortunately, this will break existing users. There is no migration, so they
will have to wipe `/var/lib/containerd` and recreate everything. However, this
is deemed worthwhile at this point, as we won't have to judge validity of the
"Readonly" field when new snapshot types are added.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-07-24 16:58:01 -07:00
Andrew Pennebaker
1d2a079f08
update to github.com/sirupsen/logrus v1.0.0
Signed-off-by: Andrew Pennebaker <apennebaker@datapipe.com>
2017-07-21 15:39:14 -07:00
Derek McGowan
a8504277cc Merge pull request #1209 from stevvooe/remove-errors
linux, linux/shim: remove error definitions
2017-07-18 19:18:23 -07:00
Stephen J Day
6d0bcd5aec
linux, linux/shim: remove error definitions
Since we now have a common set of error definitions, mapped to existing
error codes, we no longer need the specialized error codes used for
interaction with linux processes. The main issue was that string
matching was being used to map these to useful error codes. With this
change, we use errors defined in the `errdefs` package, which map
cleanly to GRPC error codes and are recoverable on either side of the
request.

The main focus of this PR was in removin these from the shim. We may
need follow ups to ensure error codes are preserved by the `Tasks`
service.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-07-18 15:56:49 -07:00
Kenfe-Mickael Laventure
a578730a94
Update linux events topic
This also remove the duplicate events for Task{Create,Start,Delete}

Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
2017-07-18 13:47:28 +02:00
Derek McGowan
a220fb6b1f
Fix panic in snapshotter when connection fails
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-07-17 10:55:32 -07:00
Derek McGowan
6d032b99f2
Rename CommittedAt to CreatedAt in content interface
Use "created at" terminology to be consistent with the rest
of the containerd interfaces.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-07-12 16:51:16 -07:00
Derek McGowan
a78d0bdeac
Update the content interface to return info from update
Namespace keys used by client for uncompressed

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-07-12 16:22:13 -07:00
Derek McGowan
fba7463ed3
Add labels and fileters to content
Update list content command to support filters
Add label subcommand to content in dist tool to update labels
Add uncompressed label on unpack

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-07-12 13:59:17 -07:00
Akihiro Suda
b06aab713a support using multiple snapshotters simultaneously
e.g. dist pull --snapshotter btrfs ...; ctr run --snapshotter btrfs ...
(empty string defaults for overlayfs)

Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-07-12 11:16:12 -07:00
Derek McGowan
46deddf460
Update list statuses to use filters
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-07-11 12:05:36 -07:00
Derek McGowan
4322664b88
Update task service to use metadata content store
Address feedback and fix issues

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-07-11 11:11:10 -07:00
Derek McGowan
b6d58f63a8
Support for ingest namespacing
Move content status to list statuses and add single status
to interface.
Updates API to support list statuses and status
Updates snapshot key creation to be generic

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-07-11 11:11:10 -07:00
Derek McGowan
2c9004d431
Add namespace content store in metadata
Add a metadata store for content which enforces content is
only visible inside a given namespace.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-07-11 11:11:10 -07:00
Derek McGowan
106c7504f4 Merge pull request #1151 from stevvooe/image-metadata-service
images, containers: converge metadata API conventions
2017-07-11 11:05:01 -07:00
Stephen J Day
7f4c4aecf7
images, containers: converge metadata API conventions
The primary feature we get with this PR is support for filters and
labels on the image metadata store. In the process of doing this, the
conventions for the API have been converged between containers and
images, providing a model for other services.

With images, `Put` (renamed to `Update` briefly) has been split into a
`Create` and `Update`, allowing one to control the behavior around these
operations. `Update` now includes support for masking fields at the
datastore-level across both the containers and image service. Filters
are now just string values to interpreted directly within the data
store. This should allow for some interesting future use cases in which
the datastore might use the syntax for more efficient query paths.

The containers service has been updated to follow these conventions as
closely as possible.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-07-11 10:45:12 -07:00
Michael Crosby
f39693eabe Add Post to events service
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-07-07 14:23:10 -07:00
Michael Crosby
f93bfb6233 Add Exec IDs
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-07-06 15:23:08 -07:00
Michael Crosby
a60511d5aa Use typeurl package for spec types
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-07-05 15:37:26 -07:00
Michael Crosby
3448c6bafb Merge pull request #1113 from dmcgowan/snapshot-namespaces
Snapshot namespaces
2017-06-30 16:36:40 -07:00
Stephen J Day
396d89e423
cmd/ctr, service/containers: implement container filter
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-06-30 11:49:16 -07:00
Derek McGowan
3db8adc5d7
Update plugin load and snapshot service
Allow plugins to be mapped and returned by their ID.
Add skip plugin to allow plugins to decide whether they should
be loaded.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-06-29 16:16:26 -07:00
Derek McGowan
4ba4f3a1d5
Add namespaced snapshotter implementation
The namespaced snapshotter wraps an existing snapshotter and
enforces namespace.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-06-29 16:16:26 -07:00
Michael Crosby
72831b6985 Use Any instead of []byte
This allows us to get the fully TypeURL for consuming code.

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-06-29 15:32:45 -07:00
Michael Crosby
82d0208aaa Implement options for runtime specific settings
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-06-29 15:32:45 -07:00
Derek McGowan
eedcbc64cc Merge pull request #1115 from stevvooe/consolidate-errors
errdefs: centralize error handling
2017-06-29 15:18:24 -07:00
Stephen J Day
a4fadc596b
errdefs: centralize error handling
Now that we have most of the services required for use with containerd,
it was found that common patterns were used throughout services. By
defining a central `errdefs` package, we ensure that services will map
errors to and from grpc consistently and cleanly. One can decorate an
error with as much context as necessary, using `pkg/errors` and still
have the error mapped correctly via grpc.

We make a few sacrifices. At this point, the common errors we use across
the repository all map directly to grpc error codes. While this seems
positively crazy, it actually works out quite well. The error conditions
that were specific weren't super necessary and the ones that were
necessary now simply have better context information. We lose the
ability to add new codes, but this constraint may not be a bad thing.

Effectively, as long as one uses the errors defined in `errdefs`, the
error class will be mapped correctly across the grpc boundary and
everything will be good. If you don't use those definitions, the error
maps to "unknown" and the error message is preserved.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-06-29 15:00:47 -07:00
Michael Crosby
e2d5522435 Change ListProcesses to ListPids
These rpcs only return pids []uint32 so should be named that way in
order to have other rpcs that list Processes such as Exec'd processes.

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-06-28 16:10:41 -07:00
Michael Crosby
7c8acca29a Move runtime interfaces to runtime package
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-06-28 10:10:59 -07:00
Michael Crosby
f36e0193a4 Implement task update
This allows tasks to have their resources updated as they are running.

Fixes #1067

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-06-26 16:38:49 -07:00
Kenfe-Mickaël Laventure
cd7bbf2afe Merge pull request #1083 from stevvooe/common-identifiers
identifiers: use common package for identifier validation
2017-06-23 18:29:01 -07:00
Stephen J Day
70815af652
identifiers: use common package for identifier validation
A few days ago, we added validation for namespaces. We've decided to
expand these naming rules to include containers. To facilitate this, a
common package `identifiers` now provides a common validation area.
These rules will be extended to apply to task identifiers, snapshot keys
and other areas where user-provided identifiers may be used.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-06-23 16:46:45 -07:00
Michael Crosby
990536f2cc Move shim protos into linux pkg
This moves the shim's API and protos out of the containerd services
package and into the linux runtime package. This is because the shim is
an implementation detail of the linux runtime that we have and it is not
a containerd user facing api.

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-06-23 16:21:47 -07:00
Stephen J Day
12c0daa9c9
api/types: consolidate types package
To simplify use of types, we have consolidate the packages for the mount
and descriptor protobuf types into a single Go package. We also drop the
versioning from the type packages, as these types will remain the same
between versions.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-06-23 13:50:28 -07:00
Stephen J Day
c05be46348
events: move types into service package
When using events, it was found to be fairly unwieldy with a number of
extra packages. For the most part, when interacting with the events
service, we want types of the same version of the service. This has been
accomplished by moving all events types into the events package.

In addition, several fixes to the way events are marshaled have been
included. Specifically, we defer to the protobuf type registration
system to assemble events and type urls, with a little bit sheen on top
of add a containerd.io oriented namespace.

This has resulted in much cleaner event consumption and has removed the
reliance on error prone type urls, in favor of concrete types.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-06-22 19:12:25 -07:00
Kenfe-Mickael Laventure
40ce7665bd
Update execution.getTask() to provide more fine grained error
This allow returning a more meaningful message when a request context
doesn't hold a namespace.

Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
2017-06-22 07:53:40 -07:00
Stephen J Day
12a6beaeeb
*: update import paths to use versioned services
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-06-21 18:29:06 -07:00
Stephen J Day
5380585e21
namespaces: enforce a character set for namespaces
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-06-21 17:31:25 -07:00
Michael Crosby
235869eb1f Rename execution service to tasks
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-06-21 14:39:25 -07:00
Michael Crosby
8b2cf6e8e6 Fix Wait() on process/tasks
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-06-21 13:48:24 -07:00
Evan Hazlett
0b06fa8518 use event service for task wait
Signed-off-by: Evan Hazlett <ejhazlett@gmail.com>
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-06-21 13:34:24 -07:00
Michael Crosby
94eafaab60 Update GRPC for consistency
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-06-21 13:34:24 -07:00
Michael Crosby
ca2ec3e33c Merge pull request #992 from estesp/errrbody-likes-errrrors
Use error interfaces for content/metadata
2017-06-21 11:55:11 -07:00
Kenfe-Mickael Laventure
fb5a3d2989
Rename variables holding a task from c to t
Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
2017-06-21 09:26:50 -07:00
Michael Crosby
f3d9aae6e9 Merge pull request #956 from ehazlett/events-service
Events Service
2017-06-20 13:23:19 -07:00
Evan Hazlett
ab4181649b events: be nice to Sprintf
Signed-off-by: Evan Hazlett <ejhazlett@gmail.com>
2017-06-20 16:17:22 -04:00
Evan Hazlett
935645b03a events: add protos
Signed-off-by: Evan Hazlett <ejhazlett@gmail.com>

events: update events package to include emitter and use envelope proto

Signed-off-by: Evan Hazlett <ejhazlett@gmail.com>

events: add events service

Signed-off-by: Evan Hazlett <ejhazlett@gmail.com>

events: enable events service and update ctr events to use events service

Signed-off-by: Evan Hazlett <ejhazlett@gmail.com>

event listeners

Signed-off-by: Evan Hazlett <ejhazlett@gmail.com>

events: helper func for emitting in services

Signed-off-by: Evan Hazlett <ejhazlett@gmail.com>

events: improved cli for containers and tasks

Signed-off-by: Evan Hazlett <ejhazlett@gmail.com>

create event envelope with poster

Signed-off-by: Evan Hazlett <ejhazlett@gmail.com>

events: introspect event data to use for type url

Signed-off-by: Evan Hazlett <ejhazlett@gmail.com>

events: use pb encoding; add event types

Signed-off-by: Evan Hazlett <ejhazlett@gmail.com>

events: instrument content and snapshot services with events

Signed-off-by: Evan Hazlett <ejhazlett@gmail.com>

events: instrument image service with events

Signed-off-by: Evan Hazlett <ejhazlett@gmail.com>

events: instrument namespace service with events

Signed-off-by: Evan Hazlett <ejhazlett@gmail.com>

events: add namespace support

Signed-off-by: Evan Hazlett <ejhazlett@gmail.com>

events: only send events from namespace requested from client

Signed-off-by: Evan Hazlett <ejhazlett@gmail.com>

events: switch to go-events for broadcasting

Signed-off-by: Evan Hazlett <ejhazlett@gmail.com>
2017-06-20 10:47:28 -04:00
Michael Crosby
60c08fa386 Don't log AlreadyExists errors
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-06-16 16:39:52 -07:00
Michael Crosby
94e7f8e943 Setup plugin ids and dependencies
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-06-14 16:17:20 -07:00
Derek McGowan
6bbed2c125 Merge pull request #1003 from crosbymichael/gprc-metrics
Register grpc metrics in main
2017-06-14 13:04:59 -07:00
Phil Estes
e10a9aff7d
Use error interfaces for content/metadata
These interfaces allow us to preserve both the checking of error "cause"
as well as messages returned from the gRPC API so that the client gets
full error reason instead of a default "metadata: not found" in the case
of a missing image.

Signed-off-by: Phil Estes <estesp@linux.vnet.ibm.com>
2017-06-14 15:55:08 -04:00
Michael Crosby
c2dd40c9bc Register grpc metrics in main
This needs to be registered last after all other services have been
registred.

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-06-14 11:19:58 -07:00
Kenfe-Mickael Laventure
33598cc5d3
linux: Wrap error with contextual message
Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
2017-06-14 10:21:48 -07:00
Michael Crosby
2e048ec589 Merge pull request #965 from dmcgowan/content-read-at
Update push to use copy
2017-06-12 16:19:53 -07:00
Michael Crosby
ff598449d1 Add DeleteProcess API for removing execs
We need a separate API for handing the exit status and deletion of
Exec'd processes to make sure they are properly cleaned up within the
shim and daemon.

Fixes #973

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-06-12 09:32:23 -07:00
Michael Crosby
cf602e3aac Merge pull request #979 from dmcgowan/content-upload-lock-fix
Replace lockfile with reference lock
2017-06-09 15:02:57 -07:00
Michael Crosby
c45dae71db Merge pull request #983 from tonistiigi/fix-ctx-imports
Fix context package imports
2017-06-09 11:23:49 -07:00
Tonis Tiigi
39d55cc498 Fix context package imports
Conflicting with definition of plugin.Differ

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-06-09 10:58:29 -07:00
Michael Crosby
745398b2e9 Remove tasks map from service
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-06-09 10:35:17 -07:00
Derek McGowan
1cdb010783
Replace lockfile with reference lock
Updates content service to handle lock errors and return
them to the client. The client remote handler has been
updated to retry when a resource is locked until the
resource is unlocked or the expected resource exists.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-06-08 16:07:38 -07:00
Derek McGowan
9211a1daa7
Set the remote writer ref on writer creation
Ensures that status calls to the remote writer correctly
sets the ref.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-06-07 16:40:42 -07:00
Derek McGowan
621760f10a
Add ReaderAt support to content store
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-06-07 10:43:00 -07:00
Stephen J Day
af2718b01f
namespaces: support within containerd
To support multi-tenancy, containerd allows the collection of metadata
and runtime objects within a heirarchical storage primitive known as
namespaces. Data cannot be shared across these namespaces, unless
allowed by the service. This allows multiple sets of containers to
managed without interaction between the clients that management. This
means that different users, such as SwarmKit, K8s, Docker and others can
use containerd without coordination. Through labels, one may use
namespaces as a tool for cleanly organizing the use of containerd
containers, including the metadata storage for higher level features,
such as ACLs.

Namespaces

Namespaces cross-cut all containerd operations and are communicated via
context, either within the Go context or via GRPC headers. As a general
rule, no features are tied to namespace, other than organization. This
will be maintained into the future. They are created as a side-effect of
operating on them or may be created manually. Namespaces can be labeled
for organization. They cannot be deleted unless the namespace is empty,
although we may want to make it so one can clean up the entirety of
containerd by deleting a namespace.

Most users will interface with namespaces by setting in the
context or via the `CONTAINERD_NAMESPACE` environment variable, but the
experience is mostly left to the client. For `ctr` and `dist`, we have
defined a "default" namespace that will be created up on use, but there
is nothing special about it. As part of this PR we have plumbed this
behavior through all commands, cleaning up context management along the
way.

Namespaces in Action

Namespaces can be managed with the `ctr namespaces` subcommand. They
can be created, labeled and destroyed.

A few commands can demonstrate the power of namespaces for use with
images. First, lets create a namespace:

```
$ ctr namespaces create foo mylabel=bar
$ ctr namespaces ls
NAME LABELS
foo  mylabel=bar
```

We can see that we have a namespace `foo` and it has a label. Let's pull
an image:

```
$ dist pull docker.io/library/redis:latest
docker.io/library/redis:latest: resolved       |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:548a75066f3f280eb017a6ccda34c561ccf4f25459ef8e36d6ea582b6af1decf: done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:d45bc46b48e45e8c72c41aedd2a173bcc7f1ea4084a8fcfc5251b1da2a09c0b6: done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:5b690bc4eaa6434456ceaccf9b3e42229bd2691869ba439e515b28fe1a66c009: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:a858478874d144f6bfc03ae2d4598e2942fc9994159f2872e39fae88d45bd847: done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:4cdd94354d2a873333a205a02dbb853dd763c73600e0cf64f60b4bd7ab694875: done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:10a267c67f423630f3afe5e04bbbc93d578861ddcc54283526222f3ad5e895b9: done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:c54584150374aa94b9f7c3fbd743adcff5adead7a3cf7207b0e51551ac4a5517: done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:d1f9221193a65eaf1b0afc4f1d4fbb7f0f209369d2696e1c07671668e150ed2b: done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:71c1f30d820f0457df186531dc4478967d075ba449bd3168a3e82137a47daf03: done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 0.9 s total:   0.0 B (0.0 B/s)
INFO[0000] unpacking rootfs
INFO[0000] Unpacked chain id: sha256:41719840acf0f89e761f4a97c6074b6e2c6c25e3830fcb39301496b5d36f9b51
```

Now, let's list the image:

```
$ dist images ls
REF                            TYPE  DIGEST SIZE
docker.io/library/redis:latest application/vnd.docker.distribution.manifest.v2+json sha256:548a75066f3f280eb017a6ccda34c561ccf4f25459ef8e36d6ea582b6af1decf 72.7 MiB
```

That looks normal. Let's list the images for the `foo` namespace and see
this in action:

```
$ CONTAINERD_NAMESPACE=foo dist images ls
REF TYPE DIGEST SIZE
```

Look at that! Nothing was pulled in the namespace `foo`. Let's do the
same pull:

```
$ CONTAINERD_NAMESPACE=foo dist pull docker.io/library/redis:latest
docker.io/library/redis:latest: resolved       |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:548a75066f3f280eb017a6ccda34c561ccf4f25459ef8e36d6ea582b6af1decf: done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:d45bc46b48e45e8c72c41aedd2a173bcc7f1ea4084a8fcfc5251b1da2a09c0b6: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:a858478874d144f6bfc03ae2d4598e2942fc9994159f2872e39fae88d45bd847: done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:4cdd94354d2a873333a205a02dbb853dd763c73600e0cf64f60b4bd7ab694875: done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:c54584150374aa94b9f7c3fbd743adcff5adead7a3cf7207b0e51551ac4a5517: done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:71c1f30d820f0457df186531dc4478967d075ba449bd3168a3e82137a47daf03: done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:d1f9221193a65eaf1b0afc4f1d4fbb7f0f209369d2696e1c07671668e150ed2b: done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:10a267c67f423630f3afe5e04bbbc93d578861ddcc54283526222f3ad5e895b9: done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:5b690bc4eaa6434456ceaccf9b3e42229bd2691869ba439e515b28fe1a66c009: done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 0.8 s total:   0.0 B (0.0 B/s)
INFO[0000] unpacking rootfs
INFO[0000] Unpacked chain id: sha256:41719840acf0f89e761f4a97c6074b6e2c6c25e3830fcb39301496b5d36f9b51
```

Wow, that was very snappy! Looks like we pulled that image into out
namespace but didn't have to download any new data because we are
sharing storage. Let's take a peak at the images we have in `foo`:

```
$ CONTAINERD_NAMESPACE=foo dist images ls
REF                            TYPE DIGEST SIZE
docker.io/library/redis:latest application/vnd.docker.distribution.manifest.v2+json sha256:548a75066f3f280eb017a6ccda34c561ccf4f25459ef8e36d6ea582b6af1decf 72.7 MiB
```

Now, let's remove that image from `foo`:

```
$ CONTAINERD_NAMESPACE=foo dist images rm
docker.io/library/redis:latest
```

Looks like it is gone:

```
$ CONTAINERD_NAMESPACE=foo dist images ls
REF TYPE DIGEST SIZE
```

But, as we can see, it is present in the `default` namespace:

```
$ dist images ls
REF                            TYPE DIGEST SIZE
docker.io/library/redis:latest application/vnd.docker.distribution.manifest.v2+json sha256:548a75066f3f280eb017a6ccda34c561ccf4f25459ef8e36d6ea582b6af1decf 72.7 MiB
```

What happened here? We can tell by listing the namespaces to get a
better understanding:

```
$ ctr namespaces ls
NAME    LABELS
default
foo     mylabel=bar
```

From the above, we can see that the `default` namespace was created with
the standard commands without the environment variable set. Isolating
the set of shared images while sharing the data that matters.

Since we removed the images for namespace `foo`, we can remove it now:

```
$ ctr namespaces rm foo
foo
```

However, when we try to remove the `default` namespace, we get an error:

```
$ ctr namespaces rm default
ctr: unable to delete default: rpc error: code = FailedPrecondition desc = namespace default must be empty
```

This is because we require that namespaces be empty when removed.

Caveats

- While most metadata objects are namespaced, containers and tasks may
exhibit some issues. We still need to move runtimes to namespaces and
the container metadata storage may not be fully worked out.
- Still need to migrate content store to metadata storage and namespace
the content store such that some data storage (ie images).
- Specifics of snapshot driver's relation to namespace needs to be
worked out in detail.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-06-06 13:50:33 -07:00
Stephen Day
25cc7614ae Merge pull request #961 from dmcgowan/remote-read-logic-error
service/content: fix logic error on storing extra data
2017-06-06 11:53:20 -07:00
Michael Crosby
a8c5542ba8 Add checkpoint and restore to client package
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-06-06 09:58:33 -07:00
Derek McGowan
73763b954f
service/content: fix logic error on storing extra data
Clarify logic that extra data is stored when the target
buffer is full. Existing logic allows for extra data to
be stored even when more data will be read into buffer
when the remaining space is less than what was copied
from the last receive.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-06-05 13:50:56 -07:00
Michael Crosby
bf2fee2da1 Merge pull request #909 from vburenin/diff-plugin
Make Diff/Apply plugable
2017-06-05 11:04:34 -07:00
Volodymyr Burenin
3c76a667b6 Make Diff/Apply plugable
Signed-off-by: Volodymyr Burenin <vburenin@gmail.com>
2017-06-01 18:38:33 -05:00
Michael Crosby
00734ab04a Return fifo paths from Shim
This allows attach of existing fifos to be done without any information
stored on the client side.

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-06-01 14:12:02 -07:00
Stephen J Day
7c14cbc091
metadata: merge storage into package
The implementations for the storage of metadata have been merged into a
single metadata package where they can share storage primitives and
techniques. The is a requisite for the addition of namespaces, which
will require a coordinated layout for records to be organized by
namespace.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-05-26 17:16:13 -07:00
Evan Hazlett
f42c762dc4
updates the execution service to include ContainerID in tasks
Signed-off-by: Evan Hazlett <ejhazlett@gmail.com>
2017-05-25 12:15:00 -04:00
Derek McGowan
e1ed4a2ea4
Add json storage for container storage
This is just a temporary storage solution to get
containers running on the new code.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-05-23 14:22:20 -07:00
Stephen J Day
539742881d
api/services: define the container metadata service
Working from feedback on the existing implementation, we have now
introduced a central metadata object to represent the lifecycle and pin
the resources required to implement what people today know as
containers. This includes the runtime specification and the root
filesystem snapshots. We also allow arbitrary labeling of the container.
Such provisions will bring the containerd definition of container closer
to what is expected by users.

The objects that encompass today's ContainerService, centered around the
runtime, will be known as tasks. These tasks take on the existing
lifecycle behavior of containerd's containers, which means that they are
deleted when they exit. Largely, there are no other changes except for
naming.

The `Container` object will operate purely as a metadata object. No
runtime state will be held on `Container`. It only informs the execution
service on what is required for creating tasks and the resources in use
by that container. The resources referenced by that container will be
deleted when the container is deleted, if not in use. In this sense,
users can create, list, label and delete containers in a similar way as
they do with docker today, without the complexity of runtime locks that
plagues current implementations.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-05-22 23:27:53 -07:00
Michael Crosby
d7af92e00c Move Mount into mount pkg
This moves both the Mount type and mountinfo into a single mount
package.

This also opens up the root of the repo to hold the containerd client
implementation.

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-05-22 16:41:12 -07:00
Derek McGowan
b07504c713 Merge pull request #862 from crosbymichael/checkpoint
Initial Support for Checkpoint && Restore
2017-05-22 15:51:10 -07:00
Michael Crosby
7cc1b64bd8 Add checkpoint and restore
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>

Update go-runc to 49b2a02ec1ed3e4ae52d30b54a291b75

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>

Add shim to restore creation

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>

Keep checkpoint path in service

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>

Add C/R to non-shim build

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>

Checkpoint rw and image

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>

Pause container on bind checkpoints

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>

Return dump.log in error on checkpoint failure

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>

Pause container for checkpoint

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>

Update runc to 639454475cb9c8b861cc599f8bcd5c8c790ae402

For checkpoint into to work you need runc version
639454475cb9c8b861cc599f8bcd5c8c790ae402 + and criu 3.0 as this is what
I have been testing with.

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>

Move restore behind create calls

This remove the restore RPCs in favor of providing the checkpoint
information to the `Create` calls of a container.  If provided, the
container will be created/restored from the checkpoint instead of an
existing container.

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>

Regen protos after rebase

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-05-22 15:34:45 -07:00
Stephen Day
14671353af Merge pull request #880 from dmcgowan/fix-content-reader-bug
Clear extra in content service reader after consumed
2017-05-19 12:37:32 -07:00
Derek McGowan
e0136cc83f
Clear extra in content service reader after consumed
Avoid bugs where content can mistakenly be read from
extra after an EOF is returned.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-05-19 11:37:40 -07:00
Stephen J Day
178b4275ca
api/types/container: nice enum values for status
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-05-18 17:48:20 -07:00
Evan Hazlett
70dd7fcc60 shim: Ps -> Processes
Signed-off-by: Evan Hazlett <ejhazlett@gmail.com>
2017-05-18 11:05:48 -04:00
Evan Hazlett
395d21018f PsRequest/PsResponse -> ProcessesRequest/ProcessesResponse; added Any to types.Process
Signed-off-by: Evan Hazlett <ejhazlett@gmail.com>
2017-05-18 11:05:48 -04:00
Evan Hazlett
ef158f8b5e add support to kill container process by pid
This adds support for signalling a container process by pid.

Signed-off-by: Evan Hazlett <ejhazlett@gmail.com>

make Ps more extensible

Signed-off-by: Evan Hazlett <ejhazlett@gmail.com>

ps: windows support

Signed-off-by: Evan Hazlett <ejhazlett@gmail.com>
2017-05-18 11:05:48 -04:00
Derek McGowan
3ae69c43d8
Add diff service implementation
Add snapshot subcommand to ctr for creating diffs of RW layers.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-05-16 13:48:53 -07:00
Derek McGowan
098ff94b24
Add snapshot and diff service
Remove rootfs service in place of snapshot service. Adds
diff service for extracting and creating diffs. Diff
creation is not yet implemented. This service allows
pulling or creating images without needing root access to
mount. Additionally in the future this will allow containerd
to ensure extractions happen safely in a chroot if needed.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-05-15 16:50:16 -07:00
Daniel, Dao Quang Minh
6df793797e Merge pull request #846 from crosbymichael/linux-container
Move container and runtime to plugin pkg
2017-05-11 19:07:58 -05:00
Michael Crosby
466e14aa71 Move errors to plugin pkg
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-05-11 15:25:26 -07:00
Michael Crosby
01b9f5ec67 Move Container and runtime to plugin pkg
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-05-11 15:24:12 -07:00
Akihiro Suda
2562aca1a3 new service: version
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-05-11 09:07:43 +00:00
Stephen J Day
193abed96e
content: unify provider and ingester
The split between provider and ingester was a long standing division
reflecting the client-side use cases. For the most part, we were
differentiating these for the algorithms that operate them, but it made
instantation and use of the types challenging. On the server-side, this
distinction is generally less important. This change unifies these types
and in the process we get a few benefits.

The first is that we now completely access the content store over GRPC.
This was the initial intent and we have now satisfied this goal
completely. There are a few issues around listing content and getting
status, but we resolve these with simple streaming and regexp filters.
More can probably be done to polish this but the result is clean.

Several other content-oriented methods were polished in the process of
unification. We have now properly seperated out the `Abort` method to
cancel ongoing or stalled ingest processes. We have also replaced the
`Active` method with a single status method.

The transition went extremely smoothly. Once the clients were updated to
use the new methods, every thing worked as expected on the first
compile.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-05-10 17:05:53 -07:00
Akihiro Suda
837205005f fix make vet failures, and enable make vet on CI
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-05-10 04:36:41 +00:00
Derek McGowan
b12f18919d Merge pull request #786 from stevvooe/edit-content-command
cmd/dist: add ability to edit content
2017-05-08 10:59:22 -07:00
Phil Estes
91410874e7
Add pause/unpause implementation
This adds pause and unpause to containerd's execution service and the
same commands to the `ctr` client.

Signed-off-by: Phil Estes <estesp@linux.vnet.ibm.com>
2017-05-06 16:05:37 -04:00
Stephen J Day
cec8578ff0
cmd/dist: add ability to edit content
This allows one to edit content in the content store with their favorite
editor. It is as simple as this:

```console
$ dist content edit sha256:58e1a1bb75db1b5a24a462dd5e2915277ea06438c3f105138f97eb53149673c4
```

The above will pop up your $EDITOR, where you can make changes to the content.
When you are done, save and the new version will be added to the content store.
The digest of the new content will be printed to stdout:

```console
sha256:247f30ac320db65f3314b63b908a3aeaac5813eade6cabc9198b5883b22807bc
```

We can then retrieve the content quite easily:

```console
$ dist content get sha256:247f30ac320db65f3314b63b908a3aeaac5813eade6cabc9198b5883b22807bc
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 1278,
      "digest": "sha256:4a415e3663882fbc554ee830889c68a33b3585503892cc718a4698e91ef2a526"
   },
   "annotations": {},
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 1905270,
         "digest": "sha256:627beaf3eaaff1c0bc3311d60fb933c17ad04fe377e1043d9593646d8ae3bfe1"
      }
   ]
}

```

In this case, an annotations field was added to the original manifest.

While this implementation is very simple, we can add all sorts of validation
and tooling to allow one to edit images inline. Coupled with declaring the
mediatype, we could return specific errors that can allow a user to craft
valid, working modifications to images for testing and profit.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-05-02 11:05:14 -07:00
Akihiro Suda
d6f35cb0c8 execution: properly check container.Exec return value
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-04-28 07:14:17 +00:00
Michael Crosby
efe177ed77 Add exited at to events and delete requests
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-04-13 13:55:58 -07:00