Commit Graph

196 Commits

Author SHA1 Message Date
Akihiro Suda
d8b68e3ccc
Stop using math/rand.Read and rand.Seed (deprecated in Go 1.20)
From golangci-lint:

> SA1019: rand.Read has been deprecated since Go 1.20 because it
>shouldn't be used: For almost all use cases, crypto/rand.Read is more
>appropriate. (staticcheck)

> SA1019: rand.Seed has been deprecated since Go 1.20 and an alternative
>has been available since Go 1.0: Programs that call Seed and then expect
>a specific sequence of results from the global random source (using
>functions such as Int) can be broken when a dependency changes how
>much it consumes from the global random source. To avoid such breakages,
>programs that need a specific result sequence should use
>NewRand(NewSource(seed)) to obtain a random generator that other
>packages cannot access. (staticcheck)

See also:

- https://pkg.go.dev/math/rand@go1.20#Read
- https://pkg.go.dev/math/rand@go1.20#Seed

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2023-02-16 03:50:23 +09:00
Brian Goff
13652e081e fetch: Use data from descriptor when available.
OCI added support for a `data` field in a descriptor.
This field is expected to contain the content being pointed to by the
descriptor.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2023-02-10 01:12:40 +00:00
Maksym Pavlenko
06bfcd658c Enable dupword linter
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
2023-01-03 12:47:16 -08:00
Wei Fu
6b7e237fc7 chore: use go fix to cleanup old +build buildtag
Signed-off-by: Wei Fu <fuweid89@gmail.com>
2022-12-29 14:25:14 +08:00
Kazuyoshi Kato
3fd52c852f
Merge pull request #7641 from thaJeztah/fix_usage_of_deprecated_funcs
Remove uses of deprecated go-digest.NewDigestFromHex, go-digest.Digest.Hex
2022-11-08 10:13:13 -08:00
Sebastiaan van Stijn
717aa5929d
Remove uses of deprecated go-digest.NewDigestFromHex, go-digest.Digest.Hex
Both of these were deprecated in 55f675811a,
but the format of the GoDoc comments didn't follow the correct format, which
caused them not being picked up by tools as "deprecated".

This patch updates uses in the codebase to use the alternatives.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-08 14:47:05 +01:00
Nikita Rybak
5f48e00af1 Extra documentation for content interfaces.
Fixes: #7305

Signed-off-by: Nikita Rybak <nikita.rybak@gmail.com>
2022-11-08 15:55:39 +11:00
Samuel Karp
b7a8a54141
Merge pull request #6995 from akhilerm/retry-on-reset 2022-09-28 10:36:55 -07:00
Derek McGowan
bc01f8fc05
Add reader option to local content reader at
Allows optimized copying from a local content file into another file.

Signed-off-by: Derek McGowan <derek@mcg.dev>
2022-09-20 20:50:04 -07:00
Akhil Mohan
8f4c23b69f
retry request on writer reset
when a put request is retried due to the response from registry,
the body of the request should be seekable. A dynamic pipe is added
to the body so that the content of the body can be read again.
Currently a maximum of 5 resets are allowed, above which will fail the
request. A new error ErrReset is introduced which informs that a
reset has occured and request needs to be retried.

also added tests for Copy() and push() to test the new functionality

Signed-off-by: Akhil Mohan <makhil@vmware.com>
2022-09-20 22:09:11 +05:30
Kazuyoshi Kato
9d286baedf crypto/sha256 must be imported
Otherwise, go-digest panics (panic: sha256 not available).

Signed-off-by: Kazuyoshi Kato <katokazu@amazon.com>
2022-08-17 21:43:47 +00:00
Maksym Pavlenko
ca3b9b50fe Run gofmt 1.19
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
2022-08-04 18:18:33 -07:00
Kazuyoshi Kato
e9e33f847d Copy fuzzers from github.com/cncf/cncf-fuzzing
This commit copies the fuzzers from the repository except for
containerd_import_structured_fuzzer.go.

Signed-off-by: Kazuyoshi Kato <katokazu@amazon.com>
2022-07-06 15:21:48 +00:00
Kazuyoshi Kato
1a095e18ba Rename Size_ to Size
Previouslty "Size" was reserved by protoc-gen-gogoctrd and user-generated
"Size" was automatically renamed to "Size_" to avoid conflicts.

Signed-off-by: Kazuyoshi Kato <katokazu@amazon.com>
2022-04-22 15:31:53 +00:00
Kazuyoshi Kato
237ef0de9b Remove all gogoproto extensions
This commit removes the following gogoproto extensions;

- gogoproto.nullable
- gogoproto.customename
- gogoproto.unmarshaller_all
- gogoproto.stringer_all
- gogoproto.sizer_all
- gogoproto.marshaler_all
- gogoproto.goproto_unregonized_all
- gogoproto.goproto_stringer_all
- gogoproto.goproto_getters_all

None of them are supported by Google's toolchain (see #6564).

Signed-off-by: Kazuyoshi Kato <katokazu@amazon.com>
2022-04-20 07:23:28 +00:00
Kazuyoshi Kato
88c0c7201e Consolidate gogo/protobuf dependencies under our own protobuf package
This would make gogo/protobuf migration easier.

Signed-off-by: Kazuyoshi Kato <katokazu@amazon.com>
2022-04-19 15:53:36 +00:00
Kazuyoshi Kato
80b825ca2c Remove gogoproto.stdtime
This commit removes gogoproto.stdtime, since it is not supported by
Google's official toolchain
(see https://github.com/containerd/containerd/issues/6564).

Signed-off-by: Kazuyoshi Kato <katokazu@amazon.com>
2022-04-19 13:39:30 +00:00
Maksym Pavlenko
871b6b6a9f Use testify
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
2022-04-01 18:17:58 -07:00
Kazuyoshi Kato
067611fdea Remove enumvalue_customname, goproto_enum_prefix and enum_customname
This commit removes gogoproto.enumvalue_customname,
gogoproto.goproto_enum_prefix and gogoproto.enum_customname.

All of them make proto-generated Go code more idiomatic, but we already
don't use these enums in our external-surfacing types and they are anyway
not supported by Google's official toolchain (see #6564).

Signed-off-by: Kazuyoshi Kato <katokazu@amazon.com>
2022-03-21 19:48:16 +00:00
Kazuyoshi Kato
3eeeb9429a Remove gogoproto.customtype
gogoproto.customtype is used to have go-digest.Digest instead of string.
While it is convinient, protoc-gen-go doesn't support the extension
and that blocks #6564.

Signed-off-by: Kazuyoshi Kato <katokazu@amazon.com>
2022-03-18 23:14:44 +00:00
Fu Wei
79d7df70d6
Merge pull request #6681 from Juneezee/test/t.TempDir 2022-03-16 14:54:16 +08:00
Eng Zer Jun
18ec2761c0
test: use T.TempDir to create temporary test directory
The directory created by `T.TempDir` is automatically removed when the
test and all its subtests complete.

Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-03-15 14:03:50 +08:00
Henry Wang
2e080bf491 Add shared content label to namespaces
Signed-off-by: Henry Wang <henwang@amazon.com>
2022-03-11 23:37:02 -08:00
Derek McGowan
48c7529de2
Fix incorrect error wrapped when closing ingest file
Signed-off-by: Derek McGowan <derek@mcg.dev>
2022-01-07 12:23:18 -08:00
Derek McGowan
0c2c289d4c
Fix seek error used without nil check
Signed-off-by: Derek McGowan <derek@mcg.dev>
2022-01-07 12:19:23 -08:00
haoyun
bbe46b8c43 feat: replace github.com/pkg/errors to errors
Signed-off-by: haoyun <yun.hao@daocloud.io>
Co-authored-by: zounengren <zouyee1989@gmail.com>
2022-01-07 10:27:03 +08:00
Tobias Klauser
df21828d27
content/local: use syscall.Timespec.Unix
Use the syscall method instead of repeating the type conversions for
the syscall.Stat_t Atim/Atimespec members. This also allows to drop the
//nolint: unconvert comments.

Signed-off-by: Tobias Klauser <tklauser@distanz.ch>
2022-01-03 12:37:25 +01:00
haoyun
c0d07094be feat: Errorf usage
Signed-off-by: haoyun <yun.hao@daocloud.io>
2021-12-13 14:31:53 +08:00
Shiming Zhang
9dfbcbbc48 content/local: Close the file if Seek fails
Signed-off-by: Shiming Zhang <wzshiming@foxmail.com>
2021-11-23 16:18:13 +08:00
zounengren
46be069379 close Writer after use which may leak mem
Signed-off-by: Zou Nengren <zouyee1989@gmail.com>
2021-10-12 17:25:08 +08:00
Eng Zer Jun
50da673592
refactor: move from io/ioutil to io and os package
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-09-21 09:50:38 +08:00
Derek McGowan
2458afeb13
Fix content copy to not ignore unexpected EOF
Signed-off-by: Derek McGowan <derek@mcg.dev>
2021-09-09 10:01:51 -07:00
Akihiro Suda
d3aa7ee9f0
Run go fmt with Go 1.17
The new `go fmt` adds `//go:build` lines (https://golang.org/doc/go1.17#tools).

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2021-08-22 09:31:50 +09:00
Kazuyoshi Kato
a3d6edc0ed content: return the error with its timestamp
The current timestamp is probably included in production logs, but
won't be in testing environments such as GitHub Actions.

Signed-off-by: Kazuyoshi Kato <katokazu@amazon.com>
2021-08-09 16:52:40 -07:00
Sebastiaan van Stijn
63c185da9a
content/local: inline sys.StatATimeAsTime()
The sys.StatATimeAsTime() utility is currently only used in a single place,
but because it's living in the "sys" package, also brings in other dependencies,
such as Microsoft/hcsshim.

This patch inlines the code from sys.StatATimeAsTime(), to remove that dependency.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-06-22 10:14:29 +02:00
Phil Estes
edfd8d5993
Change Wrapf of non-error to an actual error
Signed-off-by: Phil Estes <estesp@amazon.com>
2021-06-02 14:48:49 -04:00
Shiming Zhang
9fcea1d3f0 Rename atomicWrite to writeToCompletion
Signed-off-by: Shiming Zhang <wzshiming@foxmail.com>
2021-05-27 18:07:45 +08:00
Shiming Zhang
b890f056e8 Fix content.ReaderAt close
Signed-off-by: Shiming Zhang <wzshiming@foxmail.com>
2021-05-08 12:39:09 +08:00
Furkan
7d70992d33
tests: add test for adaptor
I noticed AdaptInfo missing test cases. I think this
func a bit important to test since we use this func
in other packages like content, store (walking), etc.

Signed-off-by: Furkan Turkal <furkan.turkal@trendyol.com>
Signed-off-by: Furkan <furkan.turkal@trendyol.com>
2021-05-05 17:39:39 +03:00
Shiming Zhang
9e576b8891 Optimize backoff
Signed-off-by: Shiming Zhang <wzshiming@foxmail.com>
2021-04-15 23:25:09 +08:00
Tonis Tiigi
8d8c15ca53 contentproxy: ensure grpc stream is closed on commit
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2021-03-18 22:46:49 -07:00
Derek McGowan
35eeb24a17
Fix exported comments enforcer in CI
Add comments where missing and fix incorrect comments

Signed-off-by: Derek McGowan <derek@mcg.dev>
2021-03-12 08:47:05 -08:00
Maksym Pavlenko
571cc32df6 Add OpenReader to create ReaderAt interface
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
2021-03-08 12:04:58 -08:00
Akihiro Suda
20346607b9
Merge pull request #4701 from kzys/content-store-filter 2021-01-25 22:35:45 +09:00
Muhammad Kaisar Arkhan
380b52652c Bring OpenBSD support
Signed-off-by: Muhammad Kaisar Arkhan <hi@yukiisbo.red>
2021-01-04 14:43:48 +01:00
Kazuyoshi Kato
c95d71cf60 content: include the staleness of the lock when tryLock() fails
When multiple clients are pulling the same image, we may have this
lock error. Short-lived locks are probably fine, but long-lived locks
may indicate that containerd has some issues.

Signed-off-by: Kazuyoshi Kato <katokazu@amazon.com>
2020-12-02 14:08:02 -08:00
Kazuyoshi Kato
e74ace9ad8 content: support filters on local.store#Walk()
While Walk() has been taking filter strings, it was not using the parameter.
This change actually makes the filtering work.

Signed-off-by: Kazuyoshi Kato <katokazu@amazon.com>
2020-11-11 11:17:48 -08:00
Amr Mahdi
b81917ee72 Add comments clarifying copyWithBuffer implementation
Signed-off-by: Amr Mahdi <amramahdi@gmail.com>
2020-11-03 04:25:42 +00:00
Amr Mahdi
f6834d4c0b replicate io.Copy optimizations
Signed-off-by: Amr Mahdi <amramahdi@gmail.com>
2020-10-28 05:50:14 +00:00
Amr Mahdi
289130b8a7 Improve image pull performance from http 1.1 container registries
Private registries that does not support http 2.0 such as Azure Container Registry streams back content in a max of 16KB chunks (max TLS record size). The small chunks introduce an overhead when copying the layers to the content store sine each chunk incurs the overhead of  grpc message that has to be sent to the content store.

This change reduces this overhead by buffering the chunks into 1MB chunks and only then writes a message to the content store.

Below is a per comparsion between the 2 approaches using a couple of large images that are being pulled from the docker hub (http 2.0) and a private Azure CR (http 1.1) in seconds.

image                                                   | Buffered copy | master
-------                                                 |---------------|----------
docker.io/pytorch/pytorch:latest                        |  55.63        | 58.33
docker.io/nvidia/cuda:latest                            |  72.05        | 75.98
containerdpulltest.azurecr.io/pytorch/pytorch:latest    | 61.45         | 77.1
containerdpulltest.azurecr.io/nvidia/cuda:latest        | 77.13         | 85.47

Signed-off-by: Amr Mahdi <amramahdi@gmail.com>
2020-10-26 05:05:09 +00:00