Commit Graph

224 Commits

Author SHA1 Message Date
Akihiro Suda
ff4acdc42e
metadata: add comments about Image.CreatedAt
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2023-03-08 03:23:01 +09:00
Akihiro Suda
4ced1fa69e
Merge pull request #8188 from dmcgowan/fix-streaming-gc-deadlock
Fix streaming manager deadlock on collection
2023-03-02 10:25:05 +09:00
Derek McGowan
5c6e9f83d4
Fix streaming manager deadlock on collection
Ensure that lock is released and stream is closed.

Signed-off-by: Derek McGowan <derek@mcg.dev>
2023-03-01 09:40:59 -08:00
Henry Wang
cf46d3c6fc Treat sandboxes as root gc resources and scan referenced objects
Signed-off-by: Henry Wang <henwang@amazon.com>
2023-02-27 19:28:28 +00:00
Akihiro Suda
b61988670c
go.mod: github.com/containerd/typeurl/v2 v2.1.0
Changes: https://github.com/containerd/typeurl/compare/7f6e6d160d67...v2.1.0

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2023-02-11 23:39:52 +09:00
Henry Wang
b9bd10c14e use local variable for rt when iterating collectors
Signed-off-by: Henry Wang <henwang@amazon.com>
2023-01-13 21:50:04 +00:00
Fu Wei
5fc727224e
Merge pull request #7861 from dmcgowan/cleanup-context
Add cleanup package for context management during cleanup
2023-01-05 13:18:31 +08:00
Derek McGowan
b550526ccd
Use cleanup.Background instead of context.Background for cleanup
Use the cleanup context to re-use values from the original context

Signed-off-by: Derek McGowan <derek@mcg.dev>
2023-01-04 11:22:24 -08:00
Maksym Pavlenko
06bfcd658c Enable dupword linter
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
2023-01-03 12:47:16 -08:00
Derek McGowan
2c573de6d3
Move snapshot event publishing into metadata store
Removes the snapshot event publishing from the snapshot service.

Adds an option to metadata db to add a publisher. Adds event
publishing to prepare, commit, and remove snapshot operations.
Adds remove snapshot event to garbage collection.

Signed-off-by: Derek McGowan <derek@mcg.dev>
2022-12-19 17:05:28 -08:00
Akihiro Suda
75b09ac4a7
images: support specifying SourceDateEpoch via ctx
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2022-12-12 09:02:35 +09:00
Derek McGowan
dcf5687cab
Add streaming service
Adds a service capable of streaming Any objects bi-directionally.
This can be used by services to send data, received data, or to
initiate requests from server to client.

Signed-off-by: Derek McGowan <derek@mcg.dev>
2022-11-30 12:55:56 -08:00
Sebastiaan van Stijn
eaedadbed0
replace strings.Split(N) for strings.Cut() or alternatives
Go 1.18 and up now provides a strings.Cut() which is better suited for
splitting key/value pairs (and similar constructs), and performs better:

```go
func BenchmarkSplit(b *testing.B) {
        b.ReportAllocs()
        data := []string{"12hello=world", "12hello=", "12=hello", "12hello"}
        for i := 0; i < b.N; i++ {
                for _, s := range data {
                        _ = strings.SplitN(s, "=", 2)[0]
                }
        }
}

func BenchmarkCut(b *testing.B) {
        b.ReportAllocs()
        data := []string{"12hello=world", "12hello=", "12=hello", "12hello"}
        for i := 0; i < b.N; i++ {
                for _, s := range data {
                        _, _, _ = strings.Cut(s, "=")
                }
        }
}
```

    BenchmarkSplit
    BenchmarkSplit-10            8244206               128.0 ns/op           128 B/op          4 allocs/op
    BenchmarkCut
    BenchmarkCut-10             54411998                21.80 ns/op            0 B/op          0 allocs/op

While looking at occurrences of `strings.Split()`, I also updated some for alternatives,
or added some constraints; for cases where an specific number of items is expected, I used `strings.SplitN()`
with a suitable limit. This prevents (theoretical) unlimited splits.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-07 10:02:25 +01:00
Kazuyoshi Kato
407703f092 Make checkContainerTimestamps less strict on Windows
This assertion is flaky on Windows.
Because of Go, Windows' time.Now resolution is lower than Linux.

Signed-off-by: Kazuyoshi Kato <katokazu@amazon.com>
2022-08-31 17:37:57 +00:00
Maksym Pavlenko
ca3b9b50fe Run gofmt 1.19
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
2022-08-04 18:18:33 -07:00
Maksym Pavlenko
e47c433d57 Add sandbox store helpers
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
2022-07-28 14:17:39 -07:00
Maksym Pavlenko
d97b754a5b Cleanup metadata tests
This commit replaces func returns with t.Cleanup,
which makes API and tests slightly easier to maintain.

Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
2022-06-27 14:42:22 -07:00
Derek McGowan
99e210c50a
Move metadata plugin registration to seperate package
Signed-off-by: Derek McGowan <derek@mcg.dev>
2022-06-22 17:38:41 -07:00
Derek McGowan
c4e29027d4
Merge pull request #6937 from mythi/sandbox-errors
sandbox: replace github.com/pkg/errors with native errors
2022-05-26 10:44:15 -07:00
Iceber Gu
8d95f2b599 fix comments on metadata schema
Signed-off-by: Iceber Gu <wei.cai-nat@daocloud.io>
2022-05-19 22:52:51 +08:00
Mikko Ylinen
523d069a25 sandbox: replace github.com/pkg/errors with native errors
PR #6366 implemented a tree-wide change to replace github.com/pkg/errors
to errors. The new sandbox API PR #6703 had few errors.Wrap*() leftovers
and pulled github.com/pkg/errors back. This commit replaces those
leftovers by following the pattern in #6366.

Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
2022-05-12 17:09:45 +03:00
Kazuyoshi Kato
2bfc2a587b
Merge pull request #6804 from dmcgowan/metadata-collectible-resources
Add collectible resources to metadata gc
2022-05-02 11:24:39 -07:00
Taeho Nam
b5370b0406
Fix comment for metadata/db.go
Fix comment

Signed-off-by: Taeho Nam <thn7440@gmail.com>
2022-04-28 13:32:45 +09:00
Kazuyoshi Kato
7a4f81d8ba Fix tests
Signed-off-by: Kazuyoshi Kato <katokazu@amazon.com>
2022-04-22 15:41:05 +00:00
Derek McGowan
3b82f9e33c
metadata: use resource max and end on registration
Ensure the registered resource type does not conflict
with existing resource types or over the max.

Signed-off-by: Derek McGowan <derek@mcg.dev>
2022-04-19 14:25:31 -07:00
Kazuyoshi Kato
88c0c7201e Consolidate gogo/protobuf dependencies under our own protobuf package
This would make gogo/protobuf migration easier.

Signed-off-by: Kazuyoshi Kato <katokazu@amazon.com>
2022-04-19 15:53:36 +00:00
Derek McGowan
8367f69fb5
Add collectible resources to metadata gc
Adds a registration function to metadata which allows plugins to
register resources to be garbage collected. These resources allow
defining resources types which are ephemeral and stored outside the
metadata plugin without extending it. The garbage collection of these
resources will not fail the metadata gc process if their removal fails.
These resources may be referenced by existing metadata store resources
but may not be used to reference metadata store resources for the purpose
of preventing garbage collection.

Signed-off-by: Derek McGowan <derek@mcg.dev>
2022-04-12 18:59:18 -07:00
Maksym Pavlenko
d0b32c0539 [sandbox] Migrate from gogo to Any
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
2022-04-08 13:34:50 -07:00
Maksym Pavlenko
17a2aaded3 [sandbox] Add ctr support
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
2022-04-08 13:33:48 -07:00
Maksym Pavlenko
00f7a6bf2b [sandbox] Address PR review comments
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
2022-04-08 13:33:47 -07:00
Maksym Pavlenko
d7ece87243 [sandbox] Save sandbox ID to container's store
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
2022-04-08 13:33:47 -07:00
Maksym Pavlenko
cab7d5b3d2 [sandbox] Implement metadata store
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
2022-04-08 13:33:47 -07:00
Maksym Pavlenko
871b6b6a9f Use testify
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
2022-04-01 18:17:58 -07:00
Kazuyoshi Kato
96b16b447d Use typeurl.Any instead of github.com/gogo/protobuf/types.Any
This commit upgrades github.com/containerd/typeurl to use typeurl.Any.
The interface hides gogo/protobuf/types.Any from containerd's Go client.

Signed-off-by: Kazuyoshi Kato <katokazu@amazon.com>
2022-03-24 20:50:07 +00:00
Fu Wei
79d7df70d6
Merge pull request #6681 from Juneezee/test/t.TempDir 2022-03-16 14:54:16 +08:00
Phil Estes
58bae86d8e
Merge pull request #6660 from henry118/shared-ns
Add shared content label to namespaces
2022-03-15 13:57:52 -07:00
Eng Zer Jun
18ec2761c0
test: use T.TempDir to create temporary test directory
The directory created by `T.TempDir` is automatically removed when the
test and all its subtests complete.

Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-03-15 14:03:50 +08:00
Henry Wang
b8bf504e94 Enable gosec linter for golangci-lint
`gosec` linter is able to identify issues described in #6584

e.g.

$ git revert 54e95e6b88
[gosec dfc8ca1ec] Revert "fix Implicit memory aliasing in for loop"
 2 files changed, 2 deletions(-)

$ make check
+ proto-fmt
+ check
GOGC=75 golangci-lint run
containerstore.go:192:54: G601: Implicit memory aliasing in for loop. (gosec)
		containers = append(containers, containerFromProto(&container))
		                                                   ^
image_store.go:132:42: G601: Implicit memory aliasing in for loop. (gosec)
		images = append(images, imageFromProto(&image))
		                                       ^
make: *** [check] Error 1

I also disabled following two settings which prevent the linter to show a complete list of issues.

* max-issues-per-linter (default 50)
* max-same-issues (default 3)

Furthermore enabling gosec revealed many other issues. For now I blacklisted the ones except G601.

Will create separate tasks to address them one by one moving next.

Signed-off-by: Henry Wang <henwang@amazon.com>
2022-03-14 22:50:54 +00:00
Henry Wang
2e080bf491 Add shared content label to namespaces
Signed-off-by: Henry Wang <henwang@amazon.com>
2022-03-11 23:37:02 -08:00
Kazuyoshi Kato
f048a25938 docs: add doc-comments on GC-related methods
Signed-off-by: Kazuyoshi Kato <katokazu@amazon.com>
2022-01-24 14:26:14 -08:00
Cody Roseborough
8dd36423b6 Revert "Add shared content label to namespaces"
This reverts commit e692a01926.

Signed-off-by: Cody Roseborough <cdr@amazon.com>
2022-01-12 16:38:06 -08:00
haoyun
bbe46b8c43 feat: replace github.com/pkg/errors to errors
Signed-off-by: haoyun <yun.hao@daocloud.io>
Co-authored-by: zounengren <zouyee1989@gmail.com>
2022-01-07 10:27:03 +08:00
Kazuyoshi Kato
44b28b61ff medatada: make namespaces' deletion error less cryptic
The error message was unnecessary cryptic. `snapshot-[name]` notation
was only used here and hard to understand.

Instead it should say `snapshots on "..." snapshotter`.

Signed-off-by: Kazuyoshi Kato <katokazu@amazon.com>
2021-12-13 09:28:24 -08:00
Derek McGowan
63b7e5771e
Merge pull request #5973 from Juneezee/deprecate-ioutil
refactor: move from io/ioutil to io and os package
2021-10-01 10:52:06 -07:00
Derek McGowan
b9cf0d75a9
Fix panic in metadata content writer on copy error
The `createAndCopy` function is only called when `nw.w` is nil
in order to create a new writer and prepare it. The current code
is attempting to close `nw.w` when there is a copy error. The
correct behavior would be to close the new writer and not touch `nw.w`.

Signed-off-by: Derek McGowan <derek@mcg.dev>
2021-09-23 10:29:52 -07:00
Eng Zer Jun
50da673592
refactor: move from io/ioutil to io and os package
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-09-21 09:50:38 +08:00
Cody Roseborough
e692a01926 Add shared content label to namespaces
Adds shared content labels to namespaces allowing content to be shared
between namespaces if that namespace is specifically tagged as being
sharable by adding the `containerd.io/namespace/sharable` label to the
namespace.

Signed-off-by: Cody Roseborough <cdr@amazon.com>
2021-07-28 18:49:32 +00:00
Mike Brown
014748bc04 fix invalid validation error checking
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
2021-06-03 15:58:11 -05:00
Iceber Gu
e37ddafab4 metadata: modify NewLeaseManager to return leases.Manager
Signed-off-by: Iceber Gu <wei.cai-nat@daocloud.io>
2021-05-07 21:26:17 +08:00
Kazuyoshi Kato
cb1580937a metadata: improve deleting a non-empty namespace's error message
Deleting a non-empty namespace fails with

> namespace must be empty: failed precondition

This change improves the error message by listing the types of
the objects in the namespace that prevent deletion.

Signed-off-by: Kazuyoshi Kato <katokazu@amazon.com>
2021-04-15 15:49:44 -07:00