While we need to support CRI v1alpha2, the implementation doesn't have
to be tied to gogo/protobuf.
Signed-off-by: Kazuyoshi Kato <katokazu@amazon.com>
Introduce cni-bridge-fp as CNI bridge plugin wrapper binary for CRI
testing.
With CNI `io.kubernetes.cri.pod-annotations` capability enabled, the user
can inject the failpoint setting by pod's annotation
`cniFailpointControlStateDir`, which stores each pod's failpoint setting
named by `${K8S_POD_NAMESPACE}-${K8S_POD_NAME}.json`.
When the plugin is invoked, the plugin will check the CNI_ARGS to get
the failpoint for the CNI_COMMAND from disk. For the testing, the user
can prepare setting before RunPodSandbox.
Signed-off-by: Wei Fu <fuweid89@gmail.com>
Added new runc shim binary in integration testing.
The shim is named by io.containerd.runc-fp.v1, which allows us to use
additional OCI annotation `io.containerd.runtime.v2.shim.failpoint.*` to
setup shim task API's failpoint. Since the shim can be shared with
multiple container, like what kubernetes pod does, the failpoint will be
initialized during setup the shim server. So, the following the
container's OCI failpoint's annotation will not work.
This commit also updates the ctr tool that we can use `--annotation` to
specify annotations when run container. For example:
```bash
➜ ctr run -d --runtime runc-fp.v1 \
--annotation "io.containerd.runtime.v2.shim.failpoint.Kill=1*error(sorry)" \
docker.io/library/alpine:latest testing sleep 1d
➜ ctr t ls
TASK PID STATUS
testing 147304 RUNNING
➜ ctr t kill -s SIGKILL testing
ctr: sorry: unknown
➜ ctr t kill -s SIGKILL testing
➜ sudo ctr t ls
TASK PID STATUS
testing 147304 STOPPED
```
The runc-fp.v1 shim is based on core runc.v2. We can use it to inject
failpoint during testing complicated or big transcation API, like
kubernetes PodRunPodsandbox.
Signed-off-by: Wei Fu <fuweid89@gmail.com>
We only need the `ToMan()` as part of the `bin/gen-manpages` binary, which
generates the man-pages; other binaries don't use this code, so we can
set the `urfave_cli_no_docs` build-tag to exclude `cpuguy83/md2man` and
`russross/blackfriday` (and other dependencies) from the binaries:
Before:
ls -lh bin
total 149M
-rwxr-xr-x 1 root root 49M May 27 10:12 containerd
-rwxr-xr-x 1 root root 6.1M May 27 10:13 containerd-shim
-rwxr-xr-x 1 root root 8.1M May 27 10:13 containerd-shim-runc-v1
-rwxr-xr-x 1 root root 8.2M May 27 10:13 containerd-shim-runc-v2
-rwxr-xr-x 1 root root 22M May 27 10:12 containerd-stress
-rwxr-xr-x 1 root root 26M May 27 10:11 ctr
-rwxr-xr-x 1 root root 30M May 27 10:14 gen-manpages
ls -l bin
total 151676
-rwxr-xr-x 1 root root 51280184 May 27 10:12 containerd
-rwxr-xr-x 1 root root 6332416 May 27 10:13 containerd-shim
-rwxr-xr-x 1 root root 8458240 May 27 10:13 containerd-shim-runc-v1
-rwxr-xr-x 1 root root 8536064 May 27 10:13 containerd-shim-runc-v2
-rwxr-xr-x 1 root root 22567160 May 27 10:12 containerd-stress
-rwxr-xr-x 1 root root 26873752 May 27 10:11 ctr
-rwxr-xr-x 1 root root 30508888 May 27 10:14 gen-manpages
After:
ls -lh bin
total 147M
-rwxr-xr-x 1 root root 49M May 27 10:26 containerd
-rwxr-xr-x 1 root root 6.1M May 27 10:26 containerd-shim
-rwxr-xr-x 1 root root 8.1M May 27 10:26 containerd-shim-runc-v1
-rwxr-xr-x 1 root root 8.2M May 27 10:26 containerd-shim-runc-v2
-rwxr-xr-x 1 root root 22M May 27 10:26 containerd-stress
-rwxr-xr-x 1 root root 26M May 27 10:26 ctr
-rwxr-xr-x 1 root root 30M May 27 10:27 gen-manpages
ls -l bin
total 149912
-rwxr-xr-x 1 root root 50930360 May 27 10:26 containerd
-rwxr-xr-x 1 root root 6332416 May 27 10:26 containerd-shim
-rwxr-xr-x 1 root root 8458240 May 27 10:26 containerd-shim-runc-v1
-rwxr-xr-x 1 root root 8536064 May 27 10:26 containerd-shim-runc-v2
-rwxr-xr-x 1 root root 22209144 May 27 10:26 containerd-stress
-rwxr-xr-x 1 root root 26523896 May 27 10:26 ctr
-rwxr-xr-x 1 root root 30508888 May 27 10:27 gen-manpages
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This commit removes the following gogoproto extensions;
- gogoproto.nullable
- gogoproto.customename
- gogoproto.unmarshaller_all
- gogoproto.stringer_all
- gogoproto.sizer_all
- gogoproto.marshaler_all
- gogoproto.goproto_unregonized_all
- gogoproto.goproto_stringer_all
- gogoproto.goproto_getters_all
None of them are supported by Google's toolchain (see #6564).
Signed-off-by: Kazuyoshi Kato <katokazu@amazon.com>
A recent change altered the default target that would get run for the makefile.
This ended up making a standalone 'make' invocation only build the Windows
shim and nothing else. This was affecting the CI of some other projects that
relied on 'make' building containerd, ctr, and friends.
Signed-off-by: Daniel Canter <dcanter@microsoft.com>
In prior releases we were not using this variable and instead were
self-constructing a release tar file.
This was changed in 27d7c50384
The change means the variable is being used now and is causing the
artifacts to be produced to have a different name which may break
download scripts.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This moves all the release builds into a Dockerfile which is a bit
cleaner for setting up our build environment.
Non-linux/amd64 builds are cross-compiled.
Currently onlinux linux/amd64, linux/arm64, and windows/amd64 are
supported, but is easy to add more, provided their is a cross-compile
toolchain available for it.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
- ensure that the root go.mod and the module specific go.mod have the
same `require` and `replace` directives for different dependencies.
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
In containerd 1.5.x, we introduced support for go modules by adding a
go.mod file in the root directory. This go.mod lists all the things
needed across the whole code base (with the exception of
integration/client which has its own go.mod). So when projects that
need to make calls to containerd API will pull in some code from
containerd/containerd, the `go mod` commands will add all the things
listed in the root go.mod to the projects go.mod file. This causes
some problems as the list of things needed to make a simple API call
is enormous. in effect, making a API call will pull everything that a
typical server needs as well as the root go.mod is all encompassing.
In general if we had smaller things folks could use, that will make it
easier by reducing the number of things that will end up in a consumers
go.mod file.
Now coming to a specific problem, the root containerd go.mod has various
k8s.io/* modules listed. Also kubernetes depends on containerd indirectly
via both moby/moby (working with docker maintainers seperately) and via
google/cadvisor. So when the kubernetes maintainers try to use latest
1.5.x containerd, they will see the kubernetes go.mod ending up depending
on the older version of kubernetes!
So if we can expose just the minimum things needed to make a client API
call then projects like cadvisor can adopt that instead of pulling in
the entire go.mod from containerd. Looking at the existing code in
cadvisor the minimum things needed would be the api/ directory from
containerd. Please see proof of concept here:
github.com/google/cadvisor/pull/2908
To enable that, in this PR, we add a go.mod file in api/ directory. we
split the Protobuild.yaml into two, one for just the things in api/
directory and the rest in the root directory. We adjust various targets
to build things correctly using `protobuild` and also ensure that we
end up with the same generated code as before as well. To ensure we
better take care of the various go.mod/go.sum files, we update the
existing `make vendor` and also add a new `make verify-vendor` that one
can run locally as well in the CI.
Ideally, we would have a `containerd/client` either as a standalone repo
or within `containerd/containerd` as a separate go module. but we will
start here to experiment with a standalone api go module first.
Also there are various follow ups we can do, for example @thaJeztah has
identified two tasks we could do after this PR lands:
github.com/containerd/containerd/pull/5716#discussion_r668821396
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
Commits 77374e8 and b5f530a changed handling of the `DESTDIR` and `PREFIX`
variables, and introduced a `MANDIR` variable.
However, in those commits, the variables are concatenated with a directory
separator (`/`); `$DESTDIR/$PREFIX`. The `$PREFIX` variable (and consequently,
the `MANDIR` variable) already should have a leading `/` (absolute path), so
there should be no need to add it. In addition, adding the `/`, would not allow
either an empty path to be passed (well, it would result in `//` in the path),
or for `$PREFIX` to be used with a relative path (with an empty `$PREFIX`).
This patch removes the directory separator.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The 'go run' command in manpage build stage currently doesn't know about
build tags yet. This could lead to strange effects when eg. optional deps
are switched off.
Therefore also pass the tags to the 'go run' calls.
Signed-off-by: Enrico Weigelt, metux IT consult <info@metux.net>
In the current implementation, the build tags arguments can catch up an
extra trailing whitespace. Since this parameter is passed in quotes, the
shell won't strip it, leading to the wrong tags passed.
Therefore just strip it explicitly.
Signed-off-by: Enrico Weigelt, metux IT consult <info@metux.net>
FHS mandates manpages should go to /usr/share/man (or /usr/local/share/man,
for site local installations) -- /usr/man was an fallout of some ancient
Unices that haven't been fully FHS conformant and usually just a symlink to
/usr/share/man, if existing at all.
Distros sometimes need to override this (eg. MVCC installs), therefore go
the 30+ years common practise way and introduce MANDIR environment variable.
Since it's a subdir under the prefix for general constant and machine
architecture independ data -- which in turn also sometimes wants to be
overridden by distros, it shall be derived from DATADIR variable.
Signed-off-by: Enrico Weigelt, metux IT consult <info@metux.net>
The DESTDIR environment variable is used in the wrong way: since 30+ years
it's common practise using it for specifying *temporary* target *root*
(where eg. packaging infrastructure picks up the install image), instead
of the installation *prefix* on the final target.
Fixing this by introducing PREFIX variable (with default /usr/local) and
using both variables according to the 30 years matured common practise.
That way, distro packagers and other standardized build/installation
systems can easily do correct deployments w/o individual hacks.
changes v2: removed variables not used yet
added documentation
Signed-off-by: Enrico Weigelt, metux IT consult <info@metux.net>
Make rules can be formulated more generic by using the $@ symbol, instead
of duplicating the target file name.
Just a little cleanup for better maintainability.
Signed-off-by: Enrico Weigelt, metux IT consult <info@metux.net>
Currently, the cri-integration tests do not work on Windows due to various reasons.
One of the reasons is because all the tests are using Linux-specific images.
Previous commits refactored the image pulling / usage in the cri-integration tests,
making it easier to update, and easier to configure a custom registry to pull images
with Windows support.
For Windows runs, custom registries can be created, which will also contain Windows
images, and the cri-integration tests can be configured to use those registries by
specifying the "--repo-list" argument, a YAML file which will contain an alternative
mapping of the default registries. This is similar to how E2E tests are handled for
Windows runs in Kubernetes.
Some of the tests are Skipped, as they do not pass yet on Windows.
Windows does not collect inodes used stats, thus, the tests that were expecting non-zero
inodes stats were failing.
Signed-off-by: Claudiu Belu <cbelu@cloudbasesolutions.com>
Packaging systems often want to override the install(1) command via INSTALL
environment variable, in order to do distro specific fixups (eg. splitting
out debug symbols from binaries to separate files).
Also use it for creating install target directories.
Signed-off-by: Enrico Weigelt, metux IT consult <info@metux.net>
Let install-man depend on man, so manpages are really built before
trying to install them.
Signed-off-by: Enrico Weigelt, metux IT consult <info@metux.net>
This is required for environments/build systems where a specific
go version / command needs to be used.
Signed-off-by: Enrico Weigelt, metux IT consult <info@metux.net>
Currently, the cri-integration tests do not work on Windows due to various reasons.
One of the reasons is because all the tests are using Linux-specific images. This
commit refactors the image pulling / usage in the cri-integration tests, making it
easier to update, and easier to configure the a custom registry to pull those images
from.
For Windows runs, custom registries can be created, which will also contain Windows
images, and the cri-integration tests can be configured to use those registries by
specifying the "--image-list" argument, a TOML file which will contain an alternative
mapping of the default images.
Signed-off-by: Claudiu Belu <cbelu@cloudbasesolutions.com>
This allows us to dig more details out of test runs and maintain a
better history.
For this we can use `gotestsum`, which is a utility that wraps `go test`
so that it outputs test2json (go's format) and output junit (a format
more easily imported into other systems).
The PR makes it possible to override the Makefile's use of `go test` to
use any other command tto executet the test. For CI we'll use `gotestsum
--`, where `gotestsum` expects everything after the `--` to be flags for
`go test`.
We then use environment variables to configure `gotestsum` (e.g.
`GOTESTSUM_JUNITFILE` is an env var accepted by `gotestsum`).
For cri tests, the test suite supports outputing test results to a
directory, these are in junit format already. The file is not named
properly just because the code that creates it (in ginkgo) is not
configured well. We can fix that upstream to give us a better name...
until then I'm keeping those results in a separate dir.
A second workflow is also added so the test results can be summed up and
a report added to the workflow run. The 2nd workflow is required for
this since PR runs do not have access to do some of this due to safety
reasons
(https://securitylab.github.com/research/github-actions-preventing-pwn-requests/)
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
`make vendor` did not do a `go mod tidy` after vendoring, whereas
CI does this as part of the vendor validation, causing CI to fail
if `make vendor` was used to update vendoring, without doing a
`go mod tidy` as well.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
