After running into performance issues when sending in certain kinds of
content, synchronous writes for content have been removed. Content is
still synced on commit, so this shouldn't be necessary.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
There is a bug in the windows CI that causes a time difference between
the host and the container.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
The locks now retry on the backend side to prevent clients from having
to round trip on locks that might be momentarily held. This exposed some
timing errors in the updated_at fields for content ingest, so we've had
to move that to a separate file to export the monotonic go runtime
timestamps.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
To support resumable download, the fetcher for a remote must implement
`io.Seeker`. If implemented the `content.Copy` function will detect the
seeker and begin from where the download was terminated by a previous
attempt.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
While early PoCs had download resumption working, we didn't have tests
and had not verified the behavior. With this test suite, we now are able
to show that download resumption is properly supported in the content
store. In particular, there was a bug where resuming a download would
not issue the writes to the correct offset in the file. A Seek was added
to ensure we are writing from the current ingest offset.
In this investigation, it was also discovered that using the OS/Disk
created time on files was skewed from the monotonic clock in Go's
runtime. The startedat values are now taken from the Go runtime and
written to a separate file.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Content commit is updated to take in a context, allowing
content to be committed within the same context the writer
was in. This is useful when commit may be able to use more
context to complete the action rather than creating its own.
An example of this being useful is for the metadata implementation
of content, having a context allows tests to fully create
content in one database transaction by making use of the context.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Ensure all writers are closed at end of test for content
test suite. Prevents test from leaving lingering connections
to the content store.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Add commit options which allow for setting labels on commit.
Prevents potential race between garbage collector reading labels
after commit and labels getting set.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
After some analysis, it was found that Content.Reader was generally
redudant to an io.ReaderAt. This change removes `Content.Reader` in
favor of a `Content.ReaderAt`. In general, `ReaderAt` can perform better
over interfaces with indeterminant latency because it avoids remote
state for reads. Where a reader is required, a helper is provided to
convert it into an `io.SectionReader`.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Export as a tar (Note: "-" can be used for stdout):
$ ctr images export /tmp/oci-busybox.tar docker.io/library/busybox:latest
Import a tar (Note: "-" can be used for stdin):
$ ctr images import foo/new:latest /tmp/oci-busybox.tar
Note: media types are not converted at the moment: e.g.
application/vnd.docker.image.rootfs.diff.tar.gzip
-> application/vnd.oci.image.layer.v1.tar+gzip
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
Fixes message on pull about a statuses not being found.
These statuses can just be ignored since they are no
longer valid and holding the database lock while reading
statuses is not necessary.
Closes#1231
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Because the lock on an ingest ref being held regardless of whether a
writer was in use, resuming an existing ingest proved impossible. We now
defer writer locking to the content store backend, where the lock will
be released automatically on closing the writer or on restarting
containerd.
There are still cases where a writer can be abandoned but not closed,
leaving an active ingest, but this is extremely rare and can be resolved
with a daemon restart.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Add content test suite with test for writer.
Update fs and metadata implementations to use test suite.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Update list content command to support filters
Add label subcommand to content in dist tool to update labels
Add uncompressed label on unpack
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Move content status to list statuses and add single status
to interface.
Updates API to support list statuses and status
Updates snapshot key creation to be generic
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Now that we have most of the services required for use with containerd,
it was found that common patterns were used throughout services. By
defining a central `errdefs` package, we ensure that services will map
errors to and from grpc consistently and cleanly. One can decorate an
error with as much context as necessary, using `pkg/errors` and still
have the error mapped correctly via grpc.
We make a few sacrifices. At this point, the common errors we use across
the repository all map directly to grpc error codes. While this seems
positively crazy, it actually works out quite well. The error conditions
that were specific weren't super necessary and the ones that were
necessary now simply have better context information. We lose the
ability to add new codes, but this constraint may not be a bad thing.
Effectively, as long as one uses the errors defined in `errdefs`, the
error class will be mapped correctly across the grpc boundary and
everything will be good. If you don't use those definitions, the error
maps to "unknown" and the error message is preserved.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
These interfaces allow us to preserve both the checking of error "cause"
as well as messages returned from the gRPC API so that the client gets
full error reason instead of a default "metadata: not found" in the case
of a missing image.
Signed-off-by: Phil Estes <estesp@linux.vnet.ibm.com>
Updates content service to handle lock errors and return
them to the client. The client remote handler has been
updated to retry when a resource is locked until the
resource is unlocked or the expected resource exists.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Open up content store writer with expected digest before
opening up the fetch call to the registry. Add Copy method
to content store helpers to allow content copy to take
advantage of seeking helper code.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
The split between provider and ingester was a long standing division
reflecting the client-side use cases. For the most part, we were
differentiating these for the algorithms that operate them, but it made
instantation and use of the types challenging. On the server-side, this
distinction is generally less important. This change unifies these types
and in the process we get a few benefits.
The first is that we now completely access the content store over GRPC.
This was the initial intent and we have now satisfied this goal
completely. There are a few issues around listing content and getting
status, but we resolve these with simple streaming and regexp filters.
More can probably be done to polish this but the result is clean.
Several other content-oriented methods were polished in the process of
unification. We have now properly seperated out the `Abort` method to
cancel ongoing or stalled ingest processes. We have also replaced the
`Active` method with a single status method.
The transition went extremely smoothly. Once the clients were updated to
use the new methods, every thing worked as expected on the first
compile.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
This mainly fixes Linux vs generic Unix differences, with some
differences between Darwin and Freebsd (which are close bit not
identical). Should make fixing for other Unix platforms easier.
Note there are not yet `runc` equivalents for these platforms;
my current use case is image manipulation for the `moby` tool.
However there is interest in OCI runtime ports for both platforms.
Current status is that MacOS can build and run `ctr`, `dist`
and `containerd` and some operations are supported. FreeBSD 11
still needs some more fixes to continuity for extended attributes.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
When compiling for armv7, the type is `int32` for `Timespec.Sec` and
`Timespec.Nsec` in the syscall package. When passing these values
to `time.Unix`, the values must be converted to `int64`.
Signed-off-by: Kevin Swiber <kswiber@gmail.com>
