These strings were resolved using the user information on the host
filesystem and were decreasing reproducibility.
For moby/buildkit issue 3688
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
"SA1019: tar.TypeRegA has been deprecated since Go 1.11 and an alternative has been available since Go 1.1:
Use TypeReg instead. (staticcheck)"
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
WithModTimeUpperBound sets the upper bound value of the ModTime property
of the tar entry structs.
WithSourceDateEpoch now implies WithModTimeUpperBound too,
in addition to WithWhiteoutTime.
For moby/buildkit issue 3296
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
This makes diff archives to be reproducible.
The value is expected to be passed from CLI applications via the $SOUCE_DATE_EPOCH env var.
See https://reproducible-builds.org/docs/source-date-epoch/
for the $SOURCE_DATE_EPOCH specification.
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
When unpacking a TAR archive, containerd preserves file's owner:
https://github.com/containerd/containerd/blob/main/archive/tar.go#L384
In some cases this behavior is not desired. In current implementation we
avoid `Lchown` on Windows. Another case when this should be skipped is
when using native snapshotter on darwin and running as non-root user.
This PR extracts a generic option - `WithNoSameOwner` (same as
`tar --no-same-owner`) to skip `Lchown` when its not required.
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
This code was copied from github.com/moby/moby/pkg/archive;
28842d3f09,
which got later simplified in
a5aed699cf
This patch aligns the containerd implementation with those changes, and uses
filepath.ToSlash() unconditionally on all platforms, as it's a no-op on platforms
that use a forward-slash; https://github.com/golang/go/blob/go1.19/src/path/filepath/path.go#L175-L183
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Calling link(2) with a symlink as the target will cause FreeBSD to
attempt to create a new hard link pointing to the target of the symlink
rather than a hardlink to the symlink itself. By contrast, Linux creates
a hardlink to the symlink.
Use linkat(2) instead, which accepts a flag controlling this behavior.
If linkat(2) is called with AT_SYMLINK_FOLLOW, it will behave the same
as link(2). If linkat(2) is called without AT_SYMLINK_FOLLOW, it will
behave the same as Linux's link(2) instead.
See FreeBSD's implementation of ln(1), which uses linkat(2) and controls
this behavior by way of the -P and -L flags:
3003117253/bin/ln/ln.c (L342-L343)
Signed-off-by: Samuel Karp <me@samuelkarp.com>
Before:
```
$ nerdctl pull gcr.io/kubeflow-images-public/tensorflow-1.14.0-notebook-cpu:v0.7.0
FATA[0026] failed to extract layer sha256:f8a604834d388fd3d04c26e4ed832b36b617ea8a4e0b1665b9199bd10cfcb171: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount1818823128:
lchown /var/lib/containerd/tmpmounts/containerd-mount1818823128/usr/local/bin/docker-credential-gcr: invalid argument: unknown
```
After:
```
$ nerdctl pull gcr.io/kubeflow-images-public/tensorflow-1.14.0-notebook-cpu:v0.7.0
FATA[0027] failed to extract layer sha256:f8a604834d388fd3d04c26e4ed832b36b617ea8a4e0b1665b9199bd10cfcb171: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount3521205359:
failed to Lchown "/var/lib/containerd/tmpmounts/containerd-mount3521205359/usr/local/bin/docker-credential-gcr" for UID 205001, GID 5000:
lchown /var/lib/containerd/tmpmounts/containerd-mount3521205359/usr/local/bin/docker-credential-gcr: invalid argument
(Hint: try increasing the number of subordinate IDs in /etc/subuid and /etc/subgid): unknown
```
Related to moby/moby issue 43576 but for containerd
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
On FreeBSD + zfs, stat call seem to return garbage in RDev for regular
files & folders. The value returned is large enough not to fit into
`Devmajor` & `Devminor` fields of the tar header. Fortunately, these
fields are required just for special devices.
This change
* adds a check into `setHeaderForSpecialDevice` that the
input header represents a special device. If it's not the case, we
don't set the Devmajor & Devminor fields.
* fixes the nil check on `getxattr`: it never returns nils, but rather
an empty slice instead
Signed-off-by: Artem Khramov <akhramov@pm.me>
Move to single lchmod interface mirroring other implementations.
Separate logic for freebsd which supports symlink no follow flag.
Signed-off-by: Derek McGowan <derek@mcg.dev>
This parallels the implementation of windowsDiff.Apply, including
bouncing very briefly though archive.WriteDiff and then straight back
out into Windows-specific code.
It's mostly pulling existing mechanisms from non-Windows Compare or
Windows Apply, and highlights that there's probably a lot of scope for
refactoring on top of this.
Now the export-related integration tests pass CI on Windows.
Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
applyFunc now takes an io.Reader instead of a tar.Reader because I'm
trying to mirror the API of the not-yet-exposed implementation of this
same behaviour in github.com/Microsoft/hcsshim/internal/ociwclayer,
with an eye to later moving to that implementation it is ever exposed.
Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
Dependencies may be switching to use the new `%w` formatting
option to wrap errors; switching to use `errors.Is()` makes
sure that we are still able to unwrap the error and detect the
underlying cause.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Some images like `criu` will have extra libs that it requires. This
adds lib support via LD_LIBRARY_PATH and InstallOpts
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
This adds a way for users to programatically install containerd binary
dependencies.
With runtime v2 and new shim's being built, it will be a challenge to
get those onto machines. Users would have to find the link, download,
place it in their path, yada yada yada.
With this functionality of a managed `/opt` directory, containerd can
use existing image and distribution infra. to get binarys, shims, etc
onto the system.
Configuration:
*default:* `/opt/containerd`
*containerd config:*
```toml
[plugins.opt]
path = "/opt/mypath"
```
Usage:
*code:*
```go
image, err := client.Pull(ctx, "docker.io/crosbymichael/runc:latest")
client.Install(ctx, image)
```
*ctr:*
```bash
ctr content fetch docker.io/crosbymichael/runc:latest
ctr install docker.io/crosbymichael/runc:latest
```
You can manage versions and see what is running via standard image
commands.
Images:
These images MUST be small and only contain binaries.
```Dockerfile
FROM scratch
Add runc /bin/runc
```
Containerd will only extract files in `/bin` of the image.
Later on, we can add support for `/lib`.
The code adds a service to manage an `/opt/containerd` directory and
provide that path to callers via the introspection service.
How to Test:
Delete runc from your system.
```bash
> sudo ctr run --rm docker.io/library/redis:alpine redis
ctr: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/containerd/io.containerd.runtime.v1.linux/default/redis/log.json: no such file or directory): exec: "runc": executable file not found in $PATH: unknown
> sudo ctr content fetch docker.io/crosbymichael/runc:latest
> sudo ctr install docker.io/crosbymichael/runc:latest
> sudo ctr run --rm docker.io/library/redis:alpine redis
1:C 01 Aug 15:59:52.864 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 01 Aug 15:59:52.864 # Redis version=4.0.10, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 01 Aug 15:59:52.864 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
1:M 01 Aug 15:59:52.866 # You requested maxclients of 10000 requiring at least 10032 max file descriptors.
1:M 01 Aug 15:59:52.866 # Server can't set maximum open files to 10032 because of OS error: Operation not permitted.
1:M 01 Aug 15:59:52.866 # Current maximum open files is 1024. maxclients has been reduced to 992 to compensate for low ulimit. If you need higher maxclients increase 'ulimit -n'.
1:M 01 Aug 15:59:52.870 * Running mode=standalone, port=6379.
1:M 01 Aug 15:59:52.870 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:M 01 Aug 15:59:52.870 # Server initialized
1:M 01 Aug 15:59:52.870 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
1:M 01 Aug 15:59:52.870 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
1:M 01 Aug 15:59:52.870 * Ready to accept connections
^C1:signal-handler (1533139193) Received SIGINT scheduling shutdown...
1:M 01 Aug 15:59:53.472 # User requested shutdown...
1:M 01 Aug 15:59:53.472 * Saving the final RDB snapshot before exiting.
1:M 01 Aug 15:59:53.484 * DB saved on disk
1:M 01 Aug 15:59:53.484 # Redis is now ready to exit, bye bye...
```
Signed-off-by: Evan Hazlett <ejhazlett@gmail.com>
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
With `fs.RootPath`, the target file will be the file which the softlink
points to, like:
touch /tmp/zzz
ln -s /tmp/zzz /tmp/xxx
ln /tmp/xxx /tmp/yyy
The `/tmp/yyy` should be same with the `/tmp/xxx`, not `/tmp/zzz`. We
should allow hardlink to softlink file.
Signed-off-by: Wei Fu <fhfuwei@163.com>
This allows non-privileged users to use containerd. This is part of a
larger track of work integrating containerd into Cloudfoundry's garden
with support for rootless.
[#156343575]
Signed-off-by: Claudia Beresford <cberesford@pivotal.io>
The go-tar implementation which is used cannot handle sockets.
There's no good reason to preserve a socket, they are basically useless without
the process which made them.
Signed-off-by: Ian Campbell <ian.campbell@docker.com>
Currently directory changes are not added to the list of
included directories, allowing those directories to receive
duplicate entries where there is both a metadata change to the
directory and a change to a file under that directory.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
To avoid buffer bloat in long running processes, we try to use buffer
pools where possible. This is meant to address shim memory usage issues,
but may not be the root cause.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Ensure whiteout is not pointing to the current directory
or parent directory before removing. Protects against invalid
removal of the parent or current directory.
Add whiteout related tar tests using manufactured tar conditions.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Call chmod on all open files and created directories to
ensure permission is set as expected without changing umask.
Fixes#1608
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
