Commit Graph

230 Commits

Author SHA1 Message Date
guiyong.ou
628f6ac681 code cleanup
Signed-off-by: guiyong.ou <guiyong.ou@daocloud.io>
2022-07-19 22:46:32 +08:00
Kazuyoshi Kato
e9e33f847d Copy fuzzers from github.com/cncf/cncf-fuzzing
This commit copies the fuzzers from the repository except for
containerd_import_structured_fuzzer.go.

Signed-off-by: Kazuyoshi Kato <katokazu@amazon.com>
2022-07-06 15:21:48 +00:00
Kazuyoshi Kato
548c9c317b Fix WWW-Authenticate parsing
According to RFC 9110, quoted-string could be "".

https://datatracker.ietf.org/doc/html/rfc9110#section-11.6.1
https://datatracker.ietf.org/doc/html/rfc9110#appendix-A

Fixes #6376.

Signed-off-by: Kazuyoshi Kato <katokazu@amazon.com>
2022-07-01 20:26:39 +00:00
Kazuyoshi Kato
b27a229df6 Move contrib/fuzz/docker_fuzzer.go to remotes/docker
Signed-off-by: Kazuyoshi Kato <katokazu@amazon.com>
2022-06-20 23:34:23 +00:00
Kazuyoshi Kato
f91616e45e Copy FuzzConvertManifest from cncf/cncf-fuzzing
This test is one of the noisiest and should be moved from
cncf/cncf-fuzzing first.

Signed-off-by: Kazuyoshi Kato <katokazu@amazon.com>
2022-06-20 23:34:23 +00:00
Akihiro Suda
42584167b7
Officially deprecate Schema 1
Schema 1 has been substantially deprecated since circa. 2017 in favor of Schema 2 introduced in Docker 1.10 (Feb 2016)
and its successor OCI Image Spec v1, but we have not officially deprecated Schema 1.

One of the reasons was that Quay did not support Schema 2 so far, but it is reported that Quay has been
supporting Schema 2 since Feb 2020 (moby/buildkit issue 409).

This PR deprecates pulling Schema 1 images but the feature will not be removed before containerd 2.0.
Pushing Schema 1 images was never implemented in containerd (and its consumers such as BuildKit).

Docker/Moby already disabled pushing Schema 1 images in Docker 20.10 (moby/moby PR 41295),
but Docker/Moby has not yet disabled pulling Schema 1 as containerd has not yet deprecated Schema 1.
(See the comments in moby/moby PR 42300.)
Docker/Moby is expected to disable pulling Schema 1 images in future after this deprecation.

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2022-05-02 19:08:38 +09:00
Fu Wei
9b33526ef6
Merge pull request #5681 from kzys/info-url
remotes/docker: log registry URLs as info instead of debug
2022-04-20 14:21:15 +08:00
Maksym Pavlenko
871b6b6a9f Use testify
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
2022-04-01 18:17:58 -07:00
Eng Zer Jun
18ec2761c0
test: use T.TempDir to create temporary test directory
The directory created by `T.TempDir` is automatically removed when the
test and all its subtests complete.

Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-03-15 14:03:50 +08:00
Fabian Hoffmann
894e780014
Improve ErrUnexpectedStatus default string
Signed-off-by: Fabian Hoffmann <fabian.hoffmann@posteo.de>
2022-03-07 09:46:25 +01:00
W3QKR2T
b90c466c38 Add ErrUnexpectedStatus to resolver
Signed-off-by: Fabian Hoffmann <fabian.hoffmann@posteo.de>
2022-03-04 10:46:39 +01:00
Shengjing Zhu
f4f41296c2 Replace golang.org/x/net/context with std library
Signed-off-by: Shengjing Zhu <zhsj@debian.org>
2022-02-22 02:27:05 +08:00
Derek McGowan
05177ab5cd
Merge pull request #6243 from ktock/pusher-abort
remotes: fix dockerPusher to handle abort correctly
2022-02-01 18:07:46 -08:00
ye.sijun
c0e00f19ab fix acr fetch token 400
Signed-off-by: ye.sijun <junnplus@gmail.com>
2022-01-27 17:34:45 +08:00
Phil Estes
ab8d99cf4b
Merge pull request #6463 from Junnplus/empty-scope
Fix empty scopes return
2022-01-20 15:34:11 -05:00
ye.sijun
936faf9c98 fix empty scopes return
Signed-off-by: ye.sijun <junnplus@gmail.com>
2022-01-20 15:16:44 +08:00
Brian Goff
f779890365 Add image handler to skip non-distributable blobs.
This makes it easy for share functionality across tools to prevent
pushing layers that are not supposed to be re-distributed.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2022-01-11 00:49:36 +00:00
Derek McGowan
8816006d1e
Fix followup items from errors replacement
Signed-off-by: Derek McGowan <derek@mcg.dev>
2022-01-07 12:16:00 -08:00
haoyun
bbe46b8c43 feat: replace github.com/pkg/errors to errors
Signed-off-by: haoyun <yun.hao@daocloud.io>
Co-authored-by: zounengren <zouyee1989@gmail.com>
2022-01-07 10:27:03 +08:00
Akihiro Suda
97623ab0cd
remotes/docker: allow fetching "refresh token" (aka "identity token")
The new AuthorizerOpt `WithFetchRefreshToken` allows fetching "refresh token"
(aka "identity token", "offline token").

For HTTP GET mode (`FetchToken`), `offline_token=true` is set in the request.
https://docs.docker.com/registry/spec/auth/token/#requesting-a-token

For HTTP POST mode (`FetchTokenWithOAuth`), `access_type=offline` is set in the request.
https://docs.docker.com/registry/spec/auth/oauth/#getting-a-token

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2021-12-27 15:19:45 +09:00
Akihiro Suda
8094f50dd0
remotes/docker/config: allow setting custom AuthorizerOpts
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2021-12-24 18:17:40 +09:00
haoyun
c0d07094be feat: Errorf usage
Signed-off-by: haoyun <yun.hao@daocloud.io>
2021-12-13 14:31:53 +08:00
Kazuyoshi Kato
5d4dc57802 remotes/docker: log registry URLs as info instead of debug
Registry URLs are important and should be logged even the logging level is
info.

Fixes #5486.

Signed-off-by: Kazuyoshi Kato <katokazu@amazon.com>
2021-12-03 10:44:05 -08:00
Kohei Tokunaga
a97564411c remotes: fix dockerPusher to handle abort correctly
`dockerPusher` provides `pushWriter` which implements `content.Writer`.
However, even if `pushWriter` become abort status (i.e. `Close()` is called
before `Commit()`), `dockerPusher` doesn't recognise that status and treats that
writer as on-going.
This behaviour doesn't allow the client to retry an aborted push.

This commit fixes this issue.
This commit also adds an test to ensure that the issue is fixed.

Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com>
2021-11-26 13:43:01 +09:00
Derek McGowan
a776a27af5
Merge pull request from GHSA-5j5w-g665-5m35
images: validate document type before unmarshal
2021-11-17 10:54:25 -08:00
Samuel Karp
70c88f5075
schema1: reject ambiguous documents
Signed-off-by: Samuel Karp <skarp@amazon.com>
2021-11-15 12:00:42 -08:00
Kohei Tokunaga
526defb571 Fix containerd fails to pull OCI image with non-http(s):// urls
Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com>
2021-11-12 09:49:28 +09:00
Jacob MacElroy
88fc5cf2d0 Adding scope tests for ParseAuthHeader
Signed-off-by: Jacob MacElroy <jacob@okteto.com>
2021-10-27 11:09:32 -06:00
Jacob MacElroy
7438edc7ed Adding tests for GenerateTokenOptions
Test are being added for GenerateTokenOptions to cover multiple scope cases and the error cases handled.

Signed-off-by: Jacob MacElroy <jacob@okteto.com>
2021-10-26 14:05:17 -06:00
Jacob MacElroy
9b4a6f1295 Generate token options with each scope as a separate string.
Currently scopes added to token options are added with all scopes included in space delimited string. This changes it so that each scope is added to the string slice as a separate string. This seems to be the desire behavior based on the fact that a string slice is used and the usage of this function in github.com/moby/buildkit.

Signed-off-by: Jacob MacElroy <jacob@okteto.com>
2021-10-26 12:57:39 -06:00
Eng Zer Jun
50da673592
refactor: move from io/ioutil to io and os package
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-09-21 09:50:38 +08:00
Phil Estes
44d5a7e26d
Merge pull request #5921 from ktock/fix-failed-precondition
Fix pull fails on unexpected EOF
2021-08-30 14:01:19 -04:00
Kohei Tokunaga
7bc5aa74c2 Fix pull fails on unexpected EOF
Currently, containerd doesn't restart pull when it encounters unexpected EOF of
blob strem withtout error codes.
There are cases where this lead to pull failure.
This commit tries to fix this issue.

Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com>
2021-08-30 11:48:43 +09:00
Akihiro Suda
d3aa7ee9f0
Run go fmt with Go 1.17
The new `go fmt` adds `//go:build` lines (https://golang.org/doc/go1.17#tools).

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2021-08-22 09:31:50 +09:00
Derek McGowan
6f027e38a8
Remove redundant build tags
Remove build tags which are already implied by the name of the file.
Ensures build tags are used consistently

Signed-off-by: Derek McGowan <derek@mcg.dev>
2021-08-05 22:27:46 -07:00
Phil Estes
6389fc79f0
Merge pull request #5770 from wzshiming/fix/push-docker-remote-close
remotes/docker/pusher.go: Fix missing Close() on push to docker remote
2021-07-21 10:06:41 -04:00
Shiming Zhang
53ec1abec2 remotes/docker/pusher.go: Fix missing Close()
Signed-off-by: Shiming Zhang <wzshiming@foxmail.com>
2021-07-21 11:37:08 +08:00
Shiming Zhang
67d07fe5cf remotes/docker/fetcher.go: Fix missing Close()
Signed-off-by: Shiming Zhang <wzshiming@foxmail.com>
2021-07-21 11:36:23 +08:00
coryb
894b6ae39b Fix missing Body.Close() calls on push to docker remote
Discovered this while using HTTP tracing via OpenTelemetry inside of
buildkitd, where the trace spans were not being reported for the
registry PUT http requests.  The spans are only reported on the Close
for the Body, after adding these Close calls, the spans are reported as
expected.

Signed-off-by: coryb <cbennett@netflix.com>
2021-07-11 08:14:57 -07:00
Derek McGowan
a7ad6b3be5
Add support for registry host path override
Adds support for mirrors which are non-compliant with the
OCI distribution specification but have previously mirrored
content with a namespace prefix after the API root `/v2`.

Signed-off-by: Derek McGowan <derek@mcg.dev>
2021-07-02 09:48:27 -07:00
Derek McGowan
95c708572f
Update documenation for OCI distribution 1.0
Signed-off-by: Derek McGowan <derek@mcg.dev>
2021-07-01 17:24:08 -07:00
Ethan Chen
50ad4b96c4 Fix incorrect UA used for registry authentication
Previously, containerd uses Go's default UA "Go-http-client/1.1" while authenticating with registry.
This commit changes it to the pattern like "containerd/v1.5.2" which is used for all other requests.

Signed-off-by: Ethan Chen <pov@mahou-shoujo.moe>
2021-05-26 09:45:54 +08:00
Derek McGowan
dbccd0d357
Merge pull request #5275 from haslersn/fix-4531
Try next mirror in case of non-404 errors, too
2021-05-19 07:57:57 -07:00
Derek McGowan
b1d4140a22
Update docker resolver to authorize redirects
Allows redirects to be authorized if authorization is provided
for the redirected to host. The authorization will always go
to the redirect and never to the referrer.

Signed-off-by: Derek McGowan <derek@mcg.dev>
2021-05-14 12:16:24 -07:00
ktock
ab1654d0e2 Fix PushHandler cannot push image that contains duplicated blobs
Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com>
2021-04-20 14:00:53 +09:00
Sebastian Hasler
c0e2f4b5d2 Try next mirror in case of non-404 errors, too
Signed-off-by: Sebastian Hasler <sebastian.hasler@gmx.net>
2021-04-15 12:04:07 +02:00
Mike Brown
d21fe46250 adds log for each failed host and status not found on host
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
2021-04-14 13:23:22 -05:00
Aaron Lehmann
4c1fa57194 remotes/docker: Only return "already exists" on push when the upload was successful
The `(dockerPusher).Push` method uses a `StatusTracker` to check if an
upload already happened, before repeating the upload. However, there is
no provision for failure handling. If a PUT request returns an error,
the `StatusTracker` will still see the upload as if it happened
successfully. Add a status boolean so that only successful uploads
short-circuit `Push`.

Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
2021-04-06 19:45:24 -07:00
Derek McGowan
261c107ffc
Merge pull request #5278 from mxpv/toml
Migrate TOML to github.com/pelletier/go-toml
2021-04-01 21:24:52 -07:00
Phil Estes
7ae0a60fb5
Add OCI ref.name to unique key in remotes handler
This allows a pusher to be used for more than one tag without creating a
new resolver/pusher. The current implementation checks the ref key
tracker status based on type and hash and will skip the push even if the
repository reference is unique.

Signed-off-by: Phil Estes <estesp@amazon.com>
2021-04-01 14:20:30 -04:00