Commit Graph

167 Commits

Author SHA1 Message Date
Brian Goff
17d4a1357e Propagate trace contexts to shims
This adds trace context propagation over the grpc/ttrpc calls to a shim.

It also adds the otlp plugin to the runc shim so that it will send
traces to the configured tracer (which is inherited from containerd's
config).
It doesn't look like this is adding any real overhead to the runc shim's
memory usage, however it does add 2MB to the binary size.
As such this is gated by a build tag `shim_tracing`

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2024-09-30 21:44:16 +00:00
Phil Estes
192679b059 Merge pull request #10659 from djdongjin/remove-duplicated-sandbox-conversion
Remove duplicated sandbox <-> proto conversion functions
2024-09-05 14:57:17 +00:00
Akihiro Suda
cae19b14f3 Merge pull request #10658 from darwin-containers/reorganize-mount-unmount
Reorganize mount/unmount code so it is easier to add Darwin-specific implementation
2024-09-03 01:51:24 +00:00
Jin Dong
93d6f0f921 remove duplicated sandbox to proto func
Signed-off-by: Jin Dong <djdongjin95@gmail.com>
2024-08-31 11:20:32 -07:00
Marat Radchenko
bfc1465a2c Reorganize mount/unmount code so it is easier to add Darwin-specific implementation
After these changes, in order to add Darwin bind-mount implementation, one only needs:
* Adjust HasBindMounts definition in mount.go
* Provide implementation in mount_darwin.go

There was no consensus on adding dependency on bindfs, that seems to be the only working solution for bind-mounts on Darwin as of today, in https://github.com/containerd/containerd/pull/8789, that's why the actual implementation is not added in current PR.

As a bonus, Linux FUSE-related code was moved to a separate file and possibly could be reused on FreeBSD, though this needs testing.

Signed-off-by: Marat Radchenko <marat@slonopotamus.org>
2024-08-30 15:25:06 +03:00
Wei Fu
3cd8f9734d core/mount: use ptrace instead of go:linkname
The Go runtime has started to [lock down future uses of linkname][1] since
go1.23. In the go source code, containerd project has been marked in the
comment, [hall of shame][2]. Well, the go:linkname is used to fork no-op
subprocess efficiently. However, since that comment, I would like to use
ptrace and remove go:linkname in the whole repository.

With go1.22 `go:linkname`:

```bash
$ go test -bench=.  -benchmem ./ -exec sudo
goos: linux
goarch: amd64
pkg: github.com/containerd/containerd/v2/core/mount
cpu: AMD Ryzen 7 5800H with Radeon Graphics
BenchmarkBatchRunGetUsernsFD_Concurrent1-16                 2440            533320 ns/op            1145 B/op         43 allocs/op
BenchmarkBatchRunGetUsernsFD_Concurrent10-16                 342           3661616 ns/op           11562 B/op        421 allocs/op
PASS
ok      github.com/containerd/containerd/v2/core/mount  2.983s
```

With go1.22 `ptrace`:

```bash
$ go test -bench=.  -benchmem ./ -exec sudo
goos: linux
goarch: amd64
pkg: github.com/containerd/containerd/v2/core/mount
cpu: AMD Ryzen 7 5800H with Radeon Graphics
BenchmarkBatchRunGetUsernsFD_Concurrent1-16                 1785            739557 ns/op            3948 B/op         68 allocs/op
BenchmarkBatchRunGetUsernsFD_Concurrent10-16                 328           4024300 ns/op           39601 B/op        671 allocs/op
PASS
ok      github.com/containerd/containerd/v2/core/mount  3.104s
```

With go1.23 `ptrace`:

```bash
$ go test -bench=.  -benchmem ./ -exec sudo
goos: linux
goarch: amd64
pkg: github.com/containerd/containerd/v2/core/mount
cpu: AMD Ryzen 7 5800H with Radeon Graphics
BenchmarkBatchRunGetUsernsFD_Concurrent1-16                 1815            723252 ns/op            4220 B/op         69 allocs/op
BenchmarkBatchRunGetUsernsFD_Concurrent10-16                 319           3957157 ns/op           42351 B/op        682 allocs/op
PASS
ok      github.com/containerd/containerd/v2/core/mount  3.051s
```

Diff:

The `ptrace` is slower than `go:linkname` mode. However, it's accepctable.

```
goos: linux
goarch: amd64
pkg: github.com/containerd/containerd/v2/core/mount
cpu: AMD Ryzen 7 5800H with Radeon Graphics
                                    │ go122-golinkname │             go122-ptrace              │             go123-ptrace              │
                                    │      sec/op      │    sec/op     vs base                 │    sec/op     vs base                 │
BatchRunGetUsernsFD_Concurrent1-16        533.3µ ± ∞ ¹   739.6µ ± ∞ ¹        ~ (p=1.000 n=1) ²   723.3µ ± ∞ ¹        ~ (p=1.000 n=1) ²
BatchRunGetUsernsFD_Concurrent10-16       3.662m ± ∞ ¹   4.024m ± ∞ ¹        ~ (p=1.000 n=1) ²   3.957m ± ∞ ¹        ~ (p=1.000 n=1) ²
geomean                                   1.397m         1.725m        +23.45%                   1.692m        +21.06%
¹ need >= 6 samples for confidence interval at level 0.95
² need >= 4 samples to detect a difference at alpha level 0.05

                                    │ go122-golinkname │              go122-ptrace               │              go123-ptrace               │
                                    │       B/op       │     B/op       vs base                  │     B/op       vs base                  │
BatchRunGetUsernsFD_Concurrent1-16       1.118Ki ± ∞ ¹   3.855Ki ± ∞ ¹         ~ (p=1.000 n=1) ²   4.121Ki ± ∞ ¹         ~ (p=1.000 n=1) ²
BatchRunGetUsernsFD_Concurrent10-16      11.29Ki ± ∞ ¹   38.67Ki ± ∞ ¹         ~ (p=1.000 n=1) ²   41.36Ki ± ∞ ¹         ~ (p=1.000 n=1) ²
geomean                                  3.553Ki         12.21Ki        +243.65%                   13.06Ki        +267.43%
¹ need >= 6 samples for confidence interval at level 0.95
² need >= 4 samples to detect a difference at alpha level 0.05

                                    │ go122-golinkname │             go122-ptrace             │             go123-ptrace             │
                                    │    allocs/op     │  allocs/op   vs base                 │  allocs/op   vs base                 │
BatchRunGetUsernsFD_Concurrent1-16         43.00 ± ∞ ¹   68.00 ± ∞ ¹        ~ (p=1.000 n=1) ²   69.00 ± ∞ ¹        ~ (p=1.000 n=1) ²
BatchRunGetUsernsFD_Concurrent10-16        421.0 ± ∞ ¹   671.0 ± ∞ ¹        ~ (p=1.000 n=1) ²   682.0 ± ∞ ¹        ~ (p=1.000 n=1) ²
geomean                                    134.5         213.6        +58.76%                   216.9        +61.23%
¹ need >= 6 samples for confidence interval at level 0.95
² need >= 4 samples to detect a difference at alpha level 0.05
```

[1]: <https://github.com/golang/go/issues/67401>
[2]: <https://github.com/golang/go/blob/release-branch.go1.23/src/runtime/proc.go#L4820>

Signed-off-by: Wei Fu <fuweid89@gmail.com>
2024-08-26 21:19:50 +08:00
Wei Fu
bcdf507363 core/mount: add benchmark test for GetUsernsFD
```bash
$ go test -bench=.  -benchmem ./ -exec sudo
goos: linux
goarch: amd64
pkg: github.com/containerd/containerd/v2/core/mount
cpu: AMD Ryzen 7 5800H with Radeon Graphics
BenchmarkBatchRunGetUsernsFD_Concurrent1-16                 2398            532424 ns/op            1145 B/op         43 allocs/op
BenchmarkBatchRunGetUsernsFD_Concurrent10-16                 343           3701695 ns/op           11552 B/op        421 allocs/op
PASS
ok      github.com/containerd/containerd/v2/core/mount  2.978s
```

Signed-off-by: Wei Fu <fuweid89@gmail.com>
2024-08-19 19:11:25 +08:00
Akhil Mohan
ebc47359ea use format string when using printf like commands
As per https://github.com/golang/go/issues/60529, printf like commands with
non-constant format strings and no args give an error in govet

Signed-off-by: Akhil Mohan <akhilerm@gmail.com>
2024-08-14 17:04:53 +05:30
Maksym Pavlenko
0b02e0c225 Merge pull request #7616 from swagatbora90/trace-cri-runtime
Add tracing spans to CRI runtime service apis
2024-08-09 18:24:47 +00:00
Sebastiaan van Stijn
9776047243 migrate to github.com/moby/sys/userns
Commit 8437c567d8 migrated the use of the
userns package to the github.com/moby/sys/user module.

After further discussion with maintainers, it was decided to move the
userns package to a separate module, as it has no direct relation with
"user" operations (other than having "user" in its name).

This patch migrates our code to use the new module.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2024-08-08 12:48:54 +02:00
Swagat Bora
c0cdcb34f1 Add spans to CRI runtime service and related client methods
This adds otel spans to CRI service mainly targeting mutating apis which includes:
* Sandbox apis - RunPodSandbox, StopPodSandbox, RemovePodSandbox
* Container apis - CreateContainer, StartContainer, StopContainer, RemoveContainer
* Attach, Exec and Exec Sync
* Containerd client methods: container.go, client.go, process.go and task.go

Signed-off-by: Swagat Bora <sbora@amazon.com>
2024-08-05 02:34:20 +00:00
Akihiro Suda
babfebf0a4 Merge pull request #10472 from fuweid/migrate-sandboxes-bucket
core/metadata: migrate sandboxes bucket into v1
2024-07-31 01:01:05 +00:00
Wei Fu
4cfeb7b19e core/metadata: migrate sandboxes bucket into v1
Signed-off-by: Wei Fu <fuweid89@gmail.com>
2024-07-30 21:37:35 +08:00
Sebastiaan van Stijn
8437c567d8 pkg/userns: deprecate and migrate to github.com/moby/sys/user/userns
The userns package in libcontainer was integrated into the moby/sys/user
module at commit [3778ae603c706494fd1e2c2faf83b406e38d687d][1].

This patch deprecates the containerd fork of that package, and adds it as
an alias for the moby/sys/user/userns package.

[1]: 3778ae603c

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2024-07-26 09:47:50 +02:00
Kazuyoshi Kato
c3bed76eeb Merge pull request #10343 from ambarve/fix_ordering_comment
Fix incorrect comment about ordering of parent layers
2024-07-23 18:19:35 +00:00
Samuel Karp
1e3c35bd0d Merge pull request #10488 from dcantah/avoid-realloc
Avoid potential reallocs by pre-sizing some slices
2024-07-22 05:39:19 +00:00
Danny Canter
b41bb6df73 Avoid potential reallocs by pre-sizing some slices
There's a couple spots where we know exactly how large
the destination buffer should be, so pre-size these to
avoid any reallocs to a higher capacity.

Signed-off-by: Danny Canter <danny@dcantah.dev>
2024-07-19 13:05:49 -07:00
Maksym Pavlenko
63b4688175 Use grpc.NewClient instead of deprecated ones
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
2024-07-18 15:26:02 -07:00
Akhil Mohan
300fd770a0 use typeurl funcs for marshalling anypb.Any
Signed-off-by: Akhil Mohan <akhilerm@gmail.com>
2024-07-10 22:26:27 +05:30
Fu Wei
f5f11658c5 Merge pull request #9903 from abel-von/add-update-resource-only
sandbox: Add Update API for sandbox controller
2024-07-01 15:16:21 +00:00
Maksym Pavlenko
686c36ba3c Merge pull request #10385 from dmcgowan/metadata-transactor
Use the transactor interface in metadata
2024-06-26 19:21:41 +00:00
Derek McGowan
8f9607eed5 Use the transactor interface in metadata
The boltdb instance in metadata is only used for getting transactions
and can also be overriden via the context to have a wider control of the
transaction boundary. Using the transactor interface allows callers of
metadata to have more control of the transaction lifecycle.

Since boltdb must be fsync'ed on commit, operations which perform many
database operations can be costly and slow. While providing transactor
via context can be used to group together operations, it does not
provide a way to manage the commit fsyncs more globally.

Signed-off-by: Derek McGowan <derek@mcg.dev>
2024-06-25 14:24:49 -07:00
Derek McGowan
2f1bf791b9 Cleanup metadata godoc
The godoc should properly show the formatting of the database
structure and sections explaining it.

Signed-off-by: Derek McGowan <derek@mcg.dev>
2024-06-19 20:16:57 -07:00
Derek McGowan
4770618c7b Merge pull request #10349 from thaJeztah/less_logrus
Remove some logrus imports
2024-06-17 17:42:29 +00:00
Fu Wei
0975ec0908 Merge pull request #10342 from dmcgowan/add-mutex-fallback-host
Adds a mutex to protect fallback host
2024-06-17 13:13:57 +00:00
Sebastiaan van Stijn
ed64e6503a core/mount: remove logrus import
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2024-06-17 12:40:18 +02:00
Sebastiaan van Stijn
ea8265fb1e core/transfer/local: remove logrus import
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2024-06-17 12:37:14 +02:00
Maksym Pavlenko
ab61734e3d Merge pull request #10231 from jedevc/add-get-token-span
auth: add span to FetchToken helpers
2024-06-14 19:33:52 +00:00
Amit Barve
fe5f85c7de Fix incorrect comment about ordering of parent layers
The ParentIDs array in the Snapshot type is populated in the reverse order i.e the
immediate parent is at the 0th index and the oldest parent is at the last index. It can be
seen here:
https://github.com/containerd/containerd/blob/main/core/snapshots/storage/bolt.go#L492
When applying these layers, the parent layer at the last index should be applied first and
the parent layer at the 0th index should be applied last. However, the comment above the
Snapshot type says the exact opposite thing.  This commit fixes that comment.

Signed-off-by: Amit Barve <ambarve@microsoft.com>
2024-06-14 11:48:06 -07:00
Derek McGowan
38e2f00382 Adds a mutex to protect fallback host
Race detector complains about concurrent access such as with Dispatch on
push.

Signed-off-by: Derek McGowan <derek@mcg.dev>
2024-06-14 10:43:39 -07:00
Abel Feng
15887d7efc sandbox: add update api for controller
Signed-off-by: Abel Feng <fshb1988@gmail.com>
2024-06-14 02:31:51 +00:00
Maksym Pavlenko
e840d1d9cc Merge pull request #10286 from dmcgowan/update-tls-fallback-default-ports
Allow fallback across default ports
2024-06-11 17:11:42 +00:00
Derek McGowan
d23c4b8b53 Use unix and windows specific connection error checks
Signed-off-by: Derek McGowan <derek@mcg.dev>
2024-06-05 14:39:57 -07:00
Derek McGowan
02b6c6939f Allow fallback across default ports
When no port is specified, allow falling back from 443 to 80 when
http is specified along with a TLS configuration.

Signed-off-by: Derek McGowan <derek@mcg.dev>
2024-06-05 14:39:57 -07:00
Maksym Pavlenko
34d3c17ae2 Merge pull request #10291 from ktock/push-platform-conf
Transfer: Push: Enable to specify platforms
2024-06-05 21:28:09 +00:00
Kohei Tokunaga
5611fdd4af Transfer: Push: Enable to specify platforms
Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com>
2024-06-04 10:02:13 +09:00
Maksym Pavlenko
1369112514 Merge pull request #10287 from ktock/transfer-push-dup
Transfer: Push: fix failure on pushing duplicated blobs
2024-06-04 00:11:56 +00:00
Derek McGowan
5d6a94a54c Merge pull request #10243 from dmcgowan/update-shim-manager-config
Cleanup shim manager configuration
2024-06-03 17:22:36 +00:00
Kohei Tokunaga
0e8cc91463 Transfer: Push: fix failure on pushing duplicated blobs
Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com>
2024-05-31 18:08:45 +09:00
Wei Fu
4123170a39 *: export RemoveVolatileOption for CRI image volumes
Remove volatile option when CRI prepares image volumes.

Signed-off-by: Wei Fu <fuweid89@gmail.com>
2024-05-30 09:56:37 +08:00
Fu Wei
5d2c988a5a Merge pull request #10201 from abel-von/retry-remote-sandbox-wait
sandbox: do retry for wait to remote sandbox controller
2024-05-29 00:00:29 +00:00
Sebastiaan van Stijn
446e63579c remove uses of platforms.Platform alias
Commit 3c8469a782 removed uses of the api
types.Platform type from public interfaces, instead using the type from
the OCI image spec.

For convenience, it also introduced an alias in the platforms package.
While this alias allows packages that already import containerd's
platforms package (now a separate module), it may also cause confusion
(it's not clear that it's an alias for the OCI type), and for packages
that do not depend on containerd's platforms package / module may now
be resulting in an extra dependency.

Let's remove the use of this alias, and instead use the OCI type directly.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2024-05-28 14:56:30 +02:00
Akhil Mohan
65024e6fd1 core/image: fix usage of "unknown" platform
"unknown" should not be returned as a valid platform
supported by the image

Signed-off-by: Akhil Mohan <akhilerm@gmail.com>
2024-05-23 16:35:44 +05:30
Derek McGowan
ca59fb0b41 Cleanup shim manager configuration
Keep platforms configuration on task manager and add environment config
for shims.

Signed-off-by: Derek McGowan <derek@mcg.dev>
2024-05-22 11:06:53 -07:00
Maksym Pavlenko
8b30607170 Provide runtime options in plugin info
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
2024-05-21 11:15:23 -07:00
Justin Chadwell
9831a62d72 auth: add span to FetchToken helpers
Before this, during a call to the docker resolver, we would generate
span wrappers for each HTTPRequest correctly, however, as the docker
resolver reaches out to the docker authorizer, it could create HTTP
requests (for fetching tokens) that would not be wrapped in any span.

This can result in rather confusing traces, e.g. something like:

	remotes.docker.resolver.HTTPRequest
		HTTP HEAD (fetch index, fails with 401)
	HTTP GET (fetch token)
	remotes.docker.resolver.HTTPRequest
		HTTP HEAD (fetch index)
	remotes.docker.resolver.HTTPRequest
		HTTP GET (fetch manifest)

By adding a span into the FetchToken, this trace becomes a little easier
to consume:

	remotes.docker.resolver.HTTPRequest
		HTTP HEAD (fetch index, fails with 401)
	remotes.docker.resolver.FetchToken
		HTTP GET (fetch token)
	remotes.docker.resolver.HTTPRequest
		HTTP HEAD (fetch index)
	remotes.docker.resolver.HTTPRequest
		HTTP GET (fetch manifest)

Signed-off-by: Justin Chadwell <me@jedevc.com>
2024-05-15 15:54:37 +01:00
Maksym Pavlenko
29a6ab8261 Merge pull request #10202 from dmcgowan/unpack-fetch-all
Unpack fetch all
2024-05-11 20:17:27 +00:00
Akihiro Suda
76895c4a31 Merge pull request #10200 from swagatbora90/preserve-unprivileged-flags
Preserve Unprivileged locked flags during remount of bind mounts
2024-05-10 21:21:31 +00:00
Derek McGowan
681a083fab Update unpacker to always fetch all
When a set of layers are provided to the unpacker, then the unpacker
should still fetch them regardless of whether they will be used for
unpack. The image handler filters are responsible for removing content
which is not intended to be fetched. Currently there is no way to use an
unpacker and also fetch all platforms.

Signed-off-by: Derek McGowan <derek@mcg.dev>
2024-05-09 21:59:42 -07:00
Abel Feng
58be881890 sandbox: do retry for wait to remote sandbox controller
remote sandbox controller may restart, the Wait call should be retried
if it is an grpc disconnetion error.

Signed-off-by: Abel Feng <fshb1988@gmail.com>
2024-05-10 10:18:42 +08:00