Remove rootfs service in place of snapshot service. Adds
diff service for extracting and creating diffs. Diff
creation is not yet implemented. This service allows
pulling or creating images without needing root access to
mount. Additionally in the future this will allow containerd
to ensure extractions happen safely in a chroot if needed.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
The split between provider and ingester was a long standing division
reflecting the client-side use cases. For the most part, we were
differentiating these for the algorithms that operate them, but it made
instantation and use of the types challenging. On the server-side, this
distinction is generally less important. This change unifies these types
and in the process we get a few benefits.
The first is that we now completely access the content store over GRPC.
This was the initial intent and we have now satisfied this goal
completely. There are a few issues around listing content and getting
status, but we resolve these with simple streaming and regexp filters.
More can probably be done to polish this but the result is clean.
Several other content-oriented methods were polished in the process of
unification. We have now properly seperated out the `Abort` method to
cancel ongoing or stalled ingest processes. We have also replaced the
`Active` method with a single status method.
The transition went extremely smoothly. Once the clients were updated to
use the new methods, every thing worked as expected on the first
compile.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
This adds pause and unpause to containerd's execution service and the
same commands to the `ctr` client.
Signed-off-by: Phil Estes <estesp@linux.vnet.ibm.com>
This is a first pass at the metadata required for supporting an image
store. We use a shallow approach to the problem, allowing this
component to centralize the naming. Resources for this image can then be
"snowballed" in for actual implementations. This is better understood
through example.
Let's take pull. One could register the name "docker.io/stevvooe/foo" as
pointing at a particular digest. When instructed to pull or fetch, the
system will notice that no components of that image are present locally.
It can then recursively resolve the resources for that image and fetch
them into the content store. Next time the instruction is issued, the
content will be present so no action will be taken.
Another example is preparing the rootfs. The requirements for a rootfs
can be resolved from a name. These "diff ids" will then be compared with
what is available in the snapshot manager. Any parts of the rootfs, such
as a layer, that isn't available in the snapshotter can be unpacked.
Once this process is satisified, the image will be runnable as a
container.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
The message was defined but the method was returning empty, plumb through the
result from the shim layer.
Compile tested only.
Signed-off-by: Ian Campbell <ian.campbell@docker.com>
To make restarting after failed pull less racy, we define `Truncate(size
int64) error` on `content.Writer` for the zero offset. Truncating a
writer will dump any existing data and digest state and start from the
beginning. All subsequent writes will start from the zero offset.
For the service, we support this by defining the behavior for a write
that changes the offset. To keep this narrow, we only support writes out
of order at the offset 0, which causes the writer to dump existing data
and reset the local hash.
This makes restarting failed pulls much smoother when there was a
previously encountered error and the source doesn't support arbitrary
seeks or reads at arbitrary offsets. By allowing this to be done while
holding the write lock on a ref, we can restart the full download
without causing a race condition.
Once we implement seeking on the `io.Reader` returned by the fetcher,
this will be less useful, but it is good to ensure that our protocol
properly supports this use case for when streaming is the only option.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Allow deletion of content over the GRPC interface. For now, we are going
with a model that conducts reference management outside of the content
store, in the metadata store but this design is valid either way.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
For clients which only want to know about one container this is simpler than
searching the result of execution.List.
Signed-off-by: Ian Campbell <ian.campbell@docker.com>
After implementing pull, a few changes are required to the content store
interface to make sure that the implementation works smoothly.
Specifically, we work to make sure the predeclaration path for digests
works the same between remote and local writers. Before, we were
hesitent to require the the size and digest up front, but it became
clear that having this provided significant benefit.
There are also several cleanups related to naming. We now call the
expected digest `Expected` consistently across the board and `Total` is
used to mark the expected size.
This whole effort comes together to provide a very smooth status
reporting workflow for image pull and push. This will be more obvious
when the bulk of pull code lands.
There are a few other changes to make `content.WriteBlob` more broadly
useful. In accordance with addition for predeclaring expected size when
getting a `Writer`, `WriteBlob` now supports this fully. It will also
resume downloads if provided an `io.Seeker` or `io.ReaderAt`. Coupled
with the `httpReadSeeker` from `docker/distribution`, we should only be
a lines of code away from resumable downloads.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Bring the content service into the containerd API. This allows the
content store to be coordinated in the containerd daemon with minimal
effort. For the most part, this API follows the conventions and behavior
of the existing content store implementation with a few caveats.
Specifically, we remove the object oriented transaction mechanism in
favor of a very rich `Write` call.
Pains are taken to reduce race conditions around when having multiple
writers to a single piece of content. Clients should be able to race
towards getting a write lock on a reference, then wait on each other.
For the most part, this should be generically pluggable to allow
implementations of the content store to be swapped out.
We'll follow this up with an implementation to validate the model.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Updates to the gogo/protobuf dependency are required to correctly
generate time types. We also remove an unused windows dependency.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
This is not really a service like the other rpcs that we expose so lets
change the import paths for it.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
After trying to explain the complexities of developing with protobuf, I
have now created a command that correctly calculates the import paths
for each package and runs the protobuf command.
The Makefile has been updated accordingly, expect we now no longer use
`go generate`. A new target `protos` has been defined. We alias the two,
for the lazy. We leave `go generate` in place for cases where we will
actually use `go generate`.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
The mount type is used across common GRPC services to express a deferred
access of a filesystem. Right now, they are generated by snapshotters,
but eventually, they can be passed for containers for creation at
runtime. With this flow, we can separate the generation and use of a
root container filesystem.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Use vendored-in protos in the project path rather
than expecting developers to have them in a local
path on the host. This made a generated change
in the FileDescriptor content, but everything else
matches and binaries are building properly.
Signed-off-by: Phil Estes <estesp@linux.vnet.ibm.com>
Generation is likely broken, but the Makefile linter now passes. Another
pass will have a fully working toolchain.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Replaced tags with owner. This assumes we will provide a Metadata
endpoint since we're planning on using it for distribution.
Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
This commit cuts out the structure for defining grpc services for this
project. To provide compatibility with go package generation and support
reuse, we use a single protobuf file per package and make the import
paths relative to the GOPATH.
This first pass attempts to position the Mount type as the lingua franca
of ContainerKit. The Images service will provide paths prepared for use
as a set of mounts of the container service.
We'll need to merge the container service in place with new file defined
here.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
* Sync process.State() with the matching events
Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
* Allow requesting events for a specific container
Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
* Sync container state retrieval with other events
Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
* Let containerd take care of calling runtime delete on exit
Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
* Take care of possible race in TestBusyboxTopExecTopKillInit
Signed-off-by: Kenfe-Mickael Laventure <mickael.laventure@gmail.com>
* containerd build clean on Solaris
Signed-off-by: Amit Krishnan <krish.amit@gmail.com>
* Vendor golang.org/x/sys
Signed-off-by: Amit Krishnan <krish.amit@gmail.com>
* Update grpc and protobufs dep
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
* Fix grpc error check
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
runc `events --stats` now has stable output so we don't need to bind to
libcontainer directly to get stats output for the containers.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Windows will not use containerd and its just unused code and unneed
complexity to keep it all around.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Windows is not going to use containerd because there is already a
similar implementation on windows. This removes all the windows files
because there is no reason to keep this overhead when its not going to
be used.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Prior to this patch, when list containers by "ctr containers" or
"ctr containers xxx", it will not get the proper status of conatinser(s).
That was caused by the wrong implementation of State() for structure process,
it only send a signal "0" to ping the "init" process and do nothing.
Since the OCI/runc has implemented an interface Status(), we can use that.
And I think this is more compatible with the design for containerd:
- containerd -> runtime -> fun()
Signed-off-by: Hu Keping <hukeping@huawei.com>
This will be filled by the value of pids.max, allowing clients to render
usage statistics for PIDs in a container.
Signed-off-by: Aleksa Sarai <asarai@suse.de>
Signed-off-by: John Howard <jhoward@microsoft.com>
Move process sorter to new file
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Sort containers by id
This will not be the most accurate sorting but atleast the list will be
consistent inbetween calls.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Allow runtime to be configurable via daemon start
This allows people to pass an alternate name or location to the runtime
binary to start containers.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Fix state output for containers
Return the proper state/status for a container by checking if the pid is
still alive. Also fix the cleanup handling in the shim to make sure
containers are not left behind.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Properly wait for container start
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Remove runtime files from containerd
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Update supervisor for orphaned containers
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Remove ctr/container.go back to rpc calls
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Add attach to loaded container
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Add monitor based on epoll for process exits
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Convert pids in containerd to string
This is so that we no longer care about linux or system level pids and
processes in containerd have user defined process id(pid) kinda like the
exec process ids that docker has today.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Add reaper back to containerd
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Implement list containers with new process model
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Implement restore of processes
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Add NONBLOCK to exit fifo open
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Implement tty reattach
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Fix race in exit pipe creation
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Add delete to shim
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Update shim to use pid-file and not stdout
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
