Commit Graph

214 Commits

Author SHA1 Message Date
Akihiro Suda
75b09ac4a7
images: support specifying SourceDateEpoch via ctx
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2022-12-12 09:02:35 +09:00
Derek McGowan
dcf5687cab
Add streaming service
Adds a service capable of streaming Any objects bi-directionally.
This can be used by services to send data, received data, or to
initiate requests from server to client.

Signed-off-by: Derek McGowan <derek@mcg.dev>
2022-11-30 12:55:56 -08:00
Sebastiaan van Stijn
eaedadbed0
replace strings.Split(N) for strings.Cut() or alternatives
Go 1.18 and up now provides a strings.Cut() which is better suited for
splitting key/value pairs (and similar constructs), and performs better:

```go
func BenchmarkSplit(b *testing.B) {
        b.ReportAllocs()
        data := []string{"12hello=world", "12hello=", "12=hello", "12hello"}
        for i := 0; i < b.N; i++ {
                for _, s := range data {
                        _ = strings.SplitN(s, "=", 2)[0]
                }
        }
}

func BenchmarkCut(b *testing.B) {
        b.ReportAllocs()
        data := []string{"12hello=world", "12hello=", "12=hello", "12hello"}
        for i := 0; i < b.N; i++ {
                for _, s := range data {
                        _, _, _ = strings.Cut(s, "=")
                }
        }
}
```

    BenchmarkSplit
    BenchmarkSplit-10            8244206               128.0 ns/op           128 B/op          4 allocs/op
    BenchmarkCut
    BenchmarkCut-10             54411998                21.80 ns/op            0 B/op          0 allocs/op

While looking at occurrences of `strings.Split()`, I also updated some for alternatives,
or added some constraints; for cases where an specific number of items is expected, I used `strings.SplitN()`
with a suitable limit. This prevents (theoretical) unlimited splits.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-07 10:02:25 +01:00
Kazuyoshi Kato
407703f092 Make checkContainerTimestamps less strict on Windows
This assertion is flaky on Windows.
Because of Go, Windows' time.Now resolution is lower than Linux.

Signed-off-by: Kazuyoshi Kato <katokazu@amazon.com>
2022-08-31 17:37:57 +00:00
Maksym Pavlenko
ca3b9b50fe Run gofmt 1.19
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
2022-08-04 18:18:33 -07:00
Maksym Pavlenko
e47c433d57 Add sandbox store helpers
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
2022-07-28 14:17:39 -07:00
Maksym Pavlenko
d97b754a5b Cleanup metadata tests
This commit replaces func returns with t.Cleanup,
which makes API and tests slightly easier to maintain.

Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
2022-06-27 14:42:22 -07:00
Derek McGowan
99e210c50a
Move metadata plugin registration to seperate package
Signed-off-by: Derek McGowan <derek@mcg.dev>
2022-06-22 17:38:41 -07:00
Derek McGowan
c4e29027d4
Merge pull request #6937 from mythi/sandbox-errors
sandbox: replace github.com/pkg/errors with native errors
2022-05-26 10:44:15 -07:00
Iceber Gu
8d95f2b599 fix comments on metadata schema
Signed-off-by: Iceber Gu <wei.cai-nat@daocloud.io>
2022-05-19 22:52:51 +08:00
Mikko Ylinen
523d069a25 sandbox: replace github.com/pkg/errors with native errors
PR #6366 implemented a tree-wide change to replace github.com/pkg/errors
to errors. The new sandbox API PR #6703 had few errors.Wrap*() leftovers
and pulled github.com/pkg/errors back. This commit replaces those
leftovers by following the pattern in #6366.

Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
2022-05-12 17:09:45 +03:00
Kazuyoshi Kato
2bfc2a587b
Merge pull request #6804 from dmcgowan/metadata-collectible-resources
Add collectible resources to metadata gc
2022-05-02 11:24:39 -07:00
Taeho Nam
b5370b0406
Fix comment for metadata/db.go
Fix comment

Signed-off-by: Taeho Nam <thn7440@gmail.com>
2022-04-28 13:32:45 +09:00
Kazuyoshi Kato
7a4f81d8ba Fix tests
Signed-off-by: Kazuyoshi Kato <katokazu@amazon.com>
2022-04-22 15:41:05 +00:00
Derek McGowan
3b82f9e33c
metadata: use resource max and end on registration
Ensure the registered resource type does not conflict
with existing resource types or over the max.

Signed-off-by: Derek McGowan <derek@mcg.dev>
2022-04-19 14:25:31 -07:00
Kazuyoshi Kato
88c0c7201e Consolidate gogo/protobuf dependencies under our own protobuf package
This would make gogo/protobuf migration easier.

Signed-off-by: Kazuyoshi Kato <katokazu@amazon.com>
2022-04-19 15:53:36 +00:00
Derek McGowan
8367f69fb5
Add collectible resources to metadata gc
Adds a registration function to metadata which allows plugins to
register resources to be garbage collected. These resources allow
defining resources types which are ephemeral and stored outside the
metadata plugin without extending it. The garbage collection of these
resources will not fail the metadata gc process if their removal fails.
These resources may be referenced by existing metadata store resources
but may not be used to reference metadata store resources for the purpose
of preventing garbage collection.

Signed-off-by: Derek McGowan <derek@mcg.dev>
2022-04-12 18:59:18 -07:00
Maksym Pavlenko
d0b32c0539 [sandbox] Migrate from gogo to Any
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
2022-04-08 13:34:50 -07:00
Maksym Pavlenko
17a2aaded3 [sandbox] Add ctr support
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
2022-04-08 13:33:48 -07:00
Maksym Pavlenko
00f7a6bf2b [sandbox] Address PR review comments
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
2022-04-08 13:33:47 -07:00
Maksym Pavlenko
d7ece87243 [sandbox] Save sandbox ID to container's store
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
2022-04-08 13:33:47 -07:00
Maksym Pavlenko
cab7d5b3d2 [sandbox] Implement metadata store
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
2022-04-08 13:33:47 -07:00
Maksym Pavlenko
871b6b6a9f Use testify
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
2022-04-01 18:17:58 -07:00
Kazuyoshi Kato
96b16b447d Use typeurl.Any instead of github.com/gogo/protobuf/types.Any
This commit upgrades github.com/containerd/typeurl to use typeurl.Any.
The interface hides gogo/protobuf/types.Any from containerd's Go client.

Signed-off-by: Kazuyoshi Kato <katokazu@amazon.com>
2022-03-24 20:50:07 +00:00
Fu Wei
79d7df70d6
Merge pull request #6681 from Juneezee/test/t.TempDir 2022-03-16 14:54:16 +08:00
Phil Estes
58bae86d8e
Merge pull request #6660 from henry118/shared-ns
Add shared content label to namespaces
2022-03-15 13:57:52 -07:00
Eng Zer Jun
18ec2761c0
test: use T.TempDir to create temporary test directory
The directory created by `T.TempDir` is automatically removed when the
test and all its subtests complete.

Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-03-15 14:03:50 +08:00
Henry Wang
b8bf504e94 Enable gosec linter for golangci-lint
`gosec` linter is able to identify issues described in #6584

e.g.

$ git revert 54e95e6b88
[gosec dfc8ca1ec] Revert "fix Implicit memory aliasing in for loop"
 2 files changed, 2 deletions(-)

$ make check
+ proto-fmt
+ check
GOGC=75 golangci-lint run
containerstore.go:192:54: G601: Implicit memory aliasing in for loop. (gosec)
		containers = append(containers, containerFromProto(&container))
		                                                   ^
image_store.go:132:42: G601: Implicit memory aliasing in for loop. (gosec)
		images = append(images, imageFromProto(&image))
		                                       ^
make: *** [check] Error 1

I also disabled following two settings which prevent the linter to show a complete list of issues.

* max-issues-per-linter (default 50)
* max-same-issues (default 3)

Furthermore enabling gosec revealed many other issues. For now I blacklisted the ones except G601.

Will create separate tasks to address them one by one moving next.

Signed-off-by: Henry Wang <henwang@amazon.com>
2022-03-14 22:50:54 +00:00
Henry Wang
2e080bf491 Add shared content label to namespaces
Signed-off-by: Henry Wang <henwang@amazon.com>
2022-03-11 23:37:02 -08:00
Kazuyoshi Kato
f048a25938 docs: add doc-comments on GC-related methods
Signed-off-by: Kazuyoshi Kato <katokazu@amazon.com>
2022-01-24 14:26:14 -08:00
Cody Roseborough
8dd36423b6 Revert "Add shared content label to namespaces"
This reverts commit e692a01926.

Signed-off-by: Cody Roseborough <cdr@amazon.com>
2022-01-12 16:38:06 -08:00
haoyun
bbe46b8c43 feat: replace github.com/pkg/errors to errors
Signed-off-by: haoyun <yun.hao@daocloud.io>
Co-authored-by: zounengren <zouyee1989@gmail.com>
2022-01-07 10:27:03 +08:00
Kazuyoshi Kato
44b28b61ff medatada: make namespaces' deletion error less cryptic
The error message was unnecessary cryptic. `snapshot-[name]` notation
was only used here and hard to understand.

Instead it should say `snapshots on "..." snapshotter`.

Signed-off-by: Kazuyoshi Kato <katokazu@amazon.com>
2021-12-13 09:28:24 -08:00
Derek McGowan
63b7e5771e
Merge pull request #5973 from Juneezee/deprecate-ioutil
refactor: move from io/ioutil to io and os package
2021-10-01 10:52:06 -07:00
Derek McGowan
b9cf0d75a9
Fix panic in metadata content writer on copy error
The `createAndCopy` function is only called when `nw.w` is nil
in order to create a new writer and prepare it. The current code
is attempting to close `nw.w` when there is a copy error. The
correct behavior would be to close the new writer and not touch `nw.w`.

Signed-off-by: Derek McGowan <derek@mcg.dev>
2021-09-23 10:29:52 -07:00
Eng Zer Jun
50da673592
refactor: move from io/ioutil to io and os package
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-09-21 09:50:38 +08:00
Cody Roseborough
e692a01926 Add shared content label to namespaces
Adds shared content labels to namespaces allowing content to be shared
between namespaces if that namespace is specifically tagged as being
sharable by adding the `containerd.io/namespace/sharable` label to the
namespace.

Signed-off-by: Cody Roseborough <cdr@amazon.com>
2021-07-28 18:49:32 +00:00
Mike Brown
014748bc04 fix invalid validation error checking
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
2021-06-03 15:58:11 -05:00
Iceber Gu
e37ddafab4 metadata: modify NewLeaseManager to return leases.Manager
Signed-off-by: Iceber Gu <wei.cai-nat@daocloud.io>
2021-05-07 21:26:17 +08:00
Kazuyoshi Kato
cb1580937a metadata: improve deleting a non-empty namespace's error message
Deleting a non-empty namespace fails with

> namespace must be empty: failed precondition

This change improves the error message by listing the types of
the objects in the namespace that prevent deletion.

Signed-off-by: Kazuyoshi Kato <katokazu@amazon.com>
2021-04-15 15:49:44 -07:00
Maksym Pavlenko
bb6c0c2de7 Add more bolt utils
This PR extracts out a few more bolt utils from the codebase for easier management.

Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
2021-04-11 19:32:46 -07:00
Kazuyoshi Kato
e74ace9ad8 content: support filters on local.store#Walk()
While Walk() has been taking filter strings, it was not using the parameter.
This change actually makes the filtering work.

Signed-off-by: Kazuyoshi Kato <katokazu@amazon.com>
2020-11-11 11:17:48 -08:00
Hajime Tazaki
f4741fb8c5 fix make test failure of missing sha256 package
Fixes: c50ff694 ("refactor(native): separate init from implementation")

Signed-off-by: Hajime Tazaki <thehajime@gmail.com>
2020-09-10 18:50:49 +09:00
Sebastiaan van Stijn
dc92ad6520
Replace errors.Cause() with errors.Is()
Dependencies may be switching to use the new `%w` formatting
option to wrap errors; switching to use `errors.Is()` makes
sure that we are still able to unwrap the error and detect the
underlying cause.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-05-08 14:36:45 +02:00
Derek McGowan
123af61c0b
Add Cleanup to snapshot API
Cleanup is an optional method a snapshotter may implement.
Cleanup can be used to cleanup resources after a snapshot
has been removed. This function allows a snapshotter to defer
longer resource cleanup until after snapshot removals are
completed. Adding this to the API allows proxy snapshotters
to leverage this enhancement.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2020-01-07 14:59:20 -08:00
ktock
493a36de95 Move label filter to snapshots package
Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com>
2019-12-25 12:13:08 +09:00
Derek McGowan
526c0db693
Support target snapshot references on prepare
Allows backend snapshots to bring existing snapshots into
a namespace without requiring clients to fully snapshots
when the target reference is known. Backend snapshots must
explicitly implement this functionality, it is equivalent
to sharing across namespaces and is up to the backend to
use the label when it is given or ignore it.

This enables remote snapshot functionality for a backend to
query for a target snapshot before a client has performed
any work to create that snapshot.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2019-11-25 10:07:25 -08:00
Derek McGowan
d1261b5087
Update snapshot parent filter property to always exist
The parent property may be empty but always exists.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2019-11-25 10:07:25 -08:00
Phil Estes
76aa4e546e
Use common identifiers package with less restrictive regex
Regular expressions in the namespaces package are redundant with the
pre-existing validator in the "identifiers" package; replace this custom
usage with the validator in "identifiers"

Signed-off-by: Phil Estes <estesp@linux.vnet.ibm.com>
2019-11-04 21:22:01 -05:00
Derek McGowan
66aa1d3ef6
Add snapshot walk implementations
Temporarily remove zfs and aufs until interface update

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2019-10-24 11:11:22 -07:00