Add a public const for "containerd.io/distribution.source" in `labels`
package and replace hardcoded usages.
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
OCI added support for a `data` field in a descriptor.
This field is expected to contain the content being pointed to by the
descriptor.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Before this required a full content.Store so that it could annotate
distribution sources.
With this change PushContent can be used with just a content.Provider.
If the content.Provider is also a content.Manager then distribution
sources will be updated accordingly.
This allows people to use this function with a significantly
implementation.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This makes it easy for share functionality across tools to prevent
pushing layers that are not supposed to be re-distributed.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This allows a pusher to be used for more than one tag without creating a
new resolver/pusher. The current implementation checks the ref key
tracker status based on type and hash and will skip the push even if the
repository reference is unique.
Signed-off-by: Phil Estes <estesp@amazon.com>
Avoid directly handling media types with "+" attributes,
instead handling the base and passing through the full
media type to the appropriate stream processor or decompression.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Adds a method for setting a custom media type key prefix used by the
fetch handler.
This allows both overwriting a built-in prefix (for reasons?) as well as
supplying a custom media type.
I added this because I was getting an error on `FetchHandler` when
pulling docker plugin images which have their own media type.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
With distribution source label in content store, select the longest
common prefix components as condidate mount blob source and try to push
with mount blob.
Fix#2964
Signed-off-by: Wei Fu <fuweid89@gmail.com>
We can use cross repository push feature to reuse the existing blobs in
the same registry. Before make push fast, we know where the blob comes
from.
Use the `containerd.io/distribution.source. = [,]` as label format. For
example, the blob is downloaded by the docker.io/library/busybox:latest
and the label will be
containerd.io/distribution.source.docker.io = library/busybox
If the blob is shared by different repos in the same registry, the repo
name will be appended, like:
containerd.io/distribution.source.docker.io = library/busybox,x/y
NOTE:
1. no need to apply for legacy docker image schema1.
2. the concurrent fetch actions might miss some repo names in label, but
it is ok.
3. it is optional. no need to add label if the engine only uses images
not push.
Signed-off-by: Wei Fu <fuweid89@gmail.com>
Gives clients more control of the pull process, allowing
the client to operate on a descriptor after it has been
pulled. This could be useful for filtering output or
tracking children before they dispatched to. This can
also be used to call custom unpackers to have visibility
into a pulled config in parallel to the downloads.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Adds a new platform interface for matching and comparing platforms.
This new interface allows both filtering and ordering of platforms
to support running multiple platform and choosing the best platform.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
This change allows implementations to resolve the location of the actual data
using OCI descriptor fields such as MediaType.
No OCI descriptor field is written to the store.
No change on gRPC API.
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
Updates blob writer helper to use new open and ensure
unavailable errors are always handled.
Removes duplication of unavailable handling code.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Fix issue where manifest content must always be fetched
even if it is already fully downloaded or shared locally.
Simplify children label setting and platform filtering.
Prevent getting a fetcher when content shared locally.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
- Use lease API (previoisly, GC was not supported)
- Refactored interfaces for ease of future Docker v1 importer support
For usage, please refer to `ctr images import --help`.
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
To allow concurrent pull of images of the v1 persuasion, we need to
backoff when multiple pullers are trying to operate on the same
resource. The back off logic is ported to v1 pull to match the behavior
for other images.
A little randomness is also added to the backoff to prevent thundering
herd and to reduce expected recovery time.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Add support for downloading layers with external URLs and
foreign/non-distributable mediatypes. This ensures that encountered
windows images are downloaded correctly. We still need to filter out the
extra windows resources when pulling linux, but this is a step towards
correctly supporting multi-platform images.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Content commit is updated to take in a context, allowing
content to be committed within the same context the writer
was in. This is useful when commit may be able to use more
context to complete the action rather than creating its own.
An example of this being useful is for the metadata implementation
of content, having a context allows tests to fully create
content in one database transaction by making use of the context.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
After some analysis, it was found that Content.Reader was generally
redudant to an io.ReaderAt. This change removes `Content.Reader` in
favor of a `Content.ReaderAt`. In general, `ReaderAt` can perform better
over interfaces with indeterminant latency because it avoids remote
state for reads. Where a reader is required, a helper is provided to
convert it into an `io.SectionReader`.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Now that we have most of the services required for use with containerd,
it was found that common patterns were used throughout services. By
defining a central `errdefs` package, we ensure that services will map
errors to and from grpc consistently and cleanly. One can decorate an
error with as much context as necessary, using `pkg/errors` and still
have the error mapped correctly via grpc.
We make a few sacrifices. At this point, the common errors we use across
the repository all map directly to grpc error codes. While this seems
positively crazy, it actually works out quite well. The error conditions
that were specific weren't super necessary and the ones that were
necessary now simply have better context information. We lose the
ability to add new codes, but this constraint may not be a bad thing.
Effectively, as long as one uses the errors defined in `errdefs`, the
error class will be mapped correctly across the grpc boundary and
everything will be good. If you don't use those definitions, the error
maps to "unknown" and the error message is preserved.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
This PR ensures that we can pull images with manifest lists, aka OCI
indexes. After this change, when pulling such an image, the resources
will all be available for creating the image.
Further support is required to do platform based selection for rootfs
creation, so such images may not yet be runnable. This is mostly useful
for checkpoint transfers, which use an OCI index for assembling the
component set.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Updates content service to handle lock errors and return
them to the client. The client remote handler has been
updated to retry when a resource is locked until the
resource is unlocked or the expected resource exists.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Open up content store writer with expected digest before
opening up the fetch call to the registry. Add Copy method
to content store helpers to allow content copy to take
advantage of seeking helper code.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
With this PR, we introduce the concept of image handlers. They support
walking a tree of image resource descriptors for doing various tasks
related to processing them. Handlers can be dispatched sequentially or
in parallel and can be stacked for various effects.
The main functionality we introduce here is parameterized fetch without
coupling format resolution to the process itself. Two important
handlers, `remotes.FetchHandler` and `image.ChildrenHandler` can be
composed to implement recursive fetch with full status reporting. The
approach can also be modified to filter based on platform or other
constraints, unlocking a lot of possibilities.
This also includes some light refactoring in the fetch command, in
preparation for submission of end to end pull.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
