Both pigz and igzip can be disabled via the environment variables.
If disabled, calling exec.LookPath and logging "not found" message is,
even in the debug level, doesn't make much sense.
Signed-off-by: Kazuyoshi Kato <kaz@fly.io>
Intel ISA-L is Intel's open source (BSD) library that outperforms both
gzip and pigz. This commit checks and uses igzip if available.
Signed-off-by: Kazuyoshi Kato <kaz@fly.io>
This update addresses an issue where the stat call on FreeBSD could
return -1 for regular files. This led to incorrect Devmajor and
Devminor values, which should be zero in such cases. Refer to the
discussion on this bug in the following PR:
https://github.com/containerd/containerd/pull/5991.
The code change now handles this scenario appropriately.
Signed-off-by: Artem Khramov <akhramov@pm.me>
The whiteout timestamps are no longer set to the source date epoch.
The source date epoch still applies to non-whiteout files.
Discussion happened in moby/buildkit PR 3560.
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
RunWithPrivileges() will enable privileges will lock a thread, change
privileges, and run the function passed in, within that thread. This
allows us to limit the scope in which we enable privileges and avoids
accidentally enabling privileges in threads that should never have them.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
It seems that in certain situations, like having the containerd root
and state on a file system hosted on a mounted VHDX, we need
SeSecurityPrivilege when opening a file with winio.ACCESS_SYSTEM_SECURITY.
This happens in the base layer writer in hcsshim when adding a new file.
Enabling SeSecurityPrivilege allows the containerd root to be hosted on
a vhdx.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
These strings were resolved using the user information on the host
filesystem and were decreasing reproducibility.
For moby/buildkit issue 3688
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
From golangci-lint:
> SA1019: rand.Read has been deprecated since Go 1.20 because it
>shouldn't be used: For almost all use cases, crypto/rand.Read is more
>appropriate. (staticcheck)
> SA1019: rand.Seed has been deprecated since Go 1.20 and an alternative
>has been available since Go 1.0: Programs that call Seed and then expect
>a specific sequence of results from the global random source (using
>functions such as Int) can be broken when a dependency changes how
>much it consumes from the global random source. To avoid such breakages,
>programs that need a specific result sequence should use
>NewRand(NewSource(seed)) to obtain a random generator that other
>packages cannot access. (staticcheck)
See also:
- https://pkg.go.dev/math/rand@go1.20#Read
- https://pkg.go.dev/math/rand@go1.20#Seed
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
"SA1019: tar.TypeRegA has been deprecated since Go 1.11 and an alternative has been available since Go 1.1:
Use TypeReg instead. (staticcheck)"
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
The previous test wasn't highly meaningful because 5 * (100 * Millisecond)
is smaller than a second.
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
WithModTimeUpperBound sets the upper bound value of the ModTime property
of the tar entry structs.
WithSourceDateEpoch now implies WithModTimeUpperBound too,
in addition to WithWhiteoutTime.
For moby/buildkit issue 3296
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
Remove nolint-comments that weren't hit by linters, and remove the "structcheck"
and "varcheck" linters, as they have been deprecated:
WARN [runner] The linter 'structcheck' is deprecated (since v1.49.0) due to: The owner seems to have abandoned the linter. Replaced by unused.
WARN [runner] The linter 'varcheck' is deprecated (since v1.49.0) due to: The owner seems to have abandoned the linter. Replaced by unused.
WARN [linters context] structcheck is disabled because of generics. You can track the evolution of the generics support by following the https://github.com/golangci/golangci-lint/issues/2649.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
- fix "nolint" comments to be in the correct format (`//nolint:<linters>[,<linter>`
no leading space, required colon (`:`) and linters.
- remove "nolint" comments for errcheck, which is disabled in our config.
- remove "nolint" comments that were no longer needed (nolintlint).
- where known, add a comment describing why a "nolint" was applied.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This makes diff archives to be reproducible.
The value is expected to be passed from CLI applications via the $SOUCE_DATE_EPOCH env var.
See https://reproducible-builds.org/docs/source-date-epoch/
for the $SOURCE_DATE_EPOCH specification.
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
It looks like this function was converting the time (`windows.NsecToTimespec()`),
only to convert it back (`windows.TimespecToNsec()`). This became clear when
moving the lines together:
```go
ctimespec := windows.NsecToTimespec(ctime.UnixNano())
c := windows.NsecToFiletime(windows.TimespecToNsec(ctimespec))
```
And looking at the Golang code, it looks like they're indeed the exact reverse:
```go
func TimespecToNsec(ts Timespec) int64 { return int64(ts.Sec)*1e9 + int64(ts.Nsec) }
func NsecToTimespec(nsec int64) (ts Timespec) {
ts.Sec = nsec / 1e9
ts.Nsec = nsec % 1e9
return
}
```
While modifying this code, also renaming the `e` variable to a more common `err`.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
When unpacking a TAR archive, containerd preserves file's owner:
https://github.com/containerd/containerd/blob/main/archive/tar.go#L384
In some cases this behavior is not desired. In current implementation we
avoid `Lchown` on Windows. Another case when this should be skipped is
when using native snapshotter on darwin and running as non-root user.
This PR extracts a generic option - `WithNoSameOwner` (same as
`tar --no-same-owner`) to skip `Lchown` when its not required.
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
This code was copied from github.com/moby/moby/pkg/archive;
28842d3f09,
which got later simplified in
a5aed699cf
This patch aligns the containerd implementation with those changes, and uses
filepath.ToSlash() unconditionally on all platforms, as it's a no-op on platforms
that use a forward-slash; https://github.com/golang/go/blob/go1.19/src/path/filepath/path.go#L175-L183
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
These functions were originally copied from the docker / moby repository in
4a7a8efc2d. Migrating these functions to use the
github.com/moby/sys/sequential module allows them being shared between moby,
docker/cli, and containerd, and to allow using them without importing all of sys
which also depends on hcsshim and more.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
`ioutil` has been deprecated by golang. All the code in `ioutil` just
forwards functionality to code in either the `io` or `os` packages.
See https://github.com/golang/go/pull/51961 for more info.
Signed-off-by: Jeff Widman <jeff@jeffwidman.com>
This commit copies the fuzzers from the repository except for
containerd_import_structured_fuzzer.go.
Signed-off-by: Kazuyoshi Kato <katokazu@amazon.com>
Different tar(1) implementations default to different input and output
locations when none is specified. This can include tape devices like
/dev/st0 (on Linux) or /dev/sa0 (on FreeBSD), but may be overridden by
compilation options or environment variables. Using the f option with
the special value of - instructs tar(1) to read from stdin and write to
stdout instead of the default.
Signed-off-by: Samuel Karp <me@samuelkarp.com>
Calling link(2) with a symlink as the target will cause FreeBSD to
attempt to create a new hard link pointing to the target of the symlink
rather than a hardlink to the symlink itself. By contrast, Linux creates
a hardlink to the symlink.
Use linkat(2) instead, which accepts a flag controlling this behavior.
If linkat(2) is called with AT_SYMLINK_FOLLOW, it will behave the same
as link(2). If linkat(2) is called without AT_SYMLINK_FOLLOW, it will
behave the same as Linux's link(2) instead.
See FreeBSD's implementation of ln(1), which uses linkat(2) and controls
this behavior by way of the -P and -L flags:
3003117253/bin/ln/ln.c (L342-L343)
Signed-off-by: Samuel Karp <me@samuelkarp.com>
Before:
```
$ nerdctl pull gcr.io/kubeflow-images-public/tensorflow-1.14.0-notebook-cpu:v0.7.0
FATA[0026] failed to extract layer sha256:f8a604834d388fd3d04c26e4ed832b36b617ea8a4e0b1665b9199bd10cfcb171: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount1818823128:
lchown /var/lib/containerd/tmpmounts/containerd-mount1818823128/usr/local/bin/docker-credential-gcr: invalid argument: unknown
```
After:
```
$ nerdctl pull gcr.io/kubeflow-images-public/tensorflow-1.14.0-notebook-cpu:v0.7.0
FATA[0027] failed to extract layer sha256:f8a604834d388fd3d04c26e4ed832b36b617ea8a4e0b1665b9199bd10cfcb171: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount3521205359:
failed to Lchown "/var/lib/containerd/tmpmounts/containerd-mount3521205359/usr/local/bin/docker-credential-gcr" for UID 205001, GID 5000:
lchown /var/lib/containerd/tmpmounts/containerd-mount3521205359/usr/local/bin/docker-credential-gcr: invalid argument
(Hint: try increasing the number of subordinate IDs in /etc/subuid and /etc/subgid): unknown
```
Related to moby/moby issue 43576 but for containerd
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
The directory created by `T.TempDir` is automatically removed when the
test and all its subtests complete.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
As a matter of fact, there are two frame formats defined by Zstandard: Zstandard frames and Skippable frames.
So we should probably support zstd algorithms with skippable frames.
See https://tools.ietf.org/id/draft-kucherawy-dispatch-zstd-00.html#rfc.section.2 for more details.
Signed-off-by: Da McGrady <dabkb@aol.com>
On FreeBSD + zfs, stat call seem to return garbage in RDev for regular
files & folders. The value returned is large enough not to fit into
`Devmajor` & `Devminor` fields of the tar header. Fortunately, these
fields are required just for special devices.
This change
* adds a check into `setHeaderForSpecialDevice` that the
input header represents a special device. If it's not the case, we
don't set the Devmajor & Devminor fields.
* fixes the nil check on `getxattr`: it never returns nils, but rather
an empty slice instead
Signed-off-by: Artem Khramov <akhramov@pm.me>
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Go 1.15.7 contained a security fix for CVE-2021-3115, which allowed arbitrary
code to be executed at build time when using cgo on Windows. This issue also
affects Unix users who have “.” listed explicitly in their PATH and are running
“go get” outside of a module or with module mode disabled.
This issue is not limited to the go command itself, and can also affect binaries
that use `os.Command`, `os.LookPath`, etc.
From the related blogpost (ttps://blog.golang.org/path-security):
> Are your own programs affected?
>
> If you use exec.LookPath or exec.Command in your own programs, you only need to
> be concerned if you (or your users) run your program in a directory with untrusted
> contents. If so, then a subprocess could be started using an executable from dot
> instead of from a system directory. (Again, using an executable from dot happens
> always on Windows and only with uncommon PATH settings on Unix.)
>
> If you are concerned, then we’ve published the more restricted variant of os/exec
> as golang.org/x/sys/execabs. You can use it in your program by simply replacing
This patch replaces all uses of `os/exec` with `golang.org/x/sys/execabs`. While
some uses of `os/exec` should not be problematic (e.g. part of tests), it is
probably good to be consistent, in case code gets moved around.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
