ci: upload junit formatted test results

This allows us to dig more details out of test runs and maintain a
better history.

For this we can use `gotestsum`, which is a utility that wraps `go test`
so that it outputs test2json (go's format) and output junit (a format
more easily imported into other systems).

The PR makes it possible to override the Makefile's use of `go test` to
use any other command tto executet the test. For CI we'll use `gotestsum
--`, where `gotestsum` expects everything after the `--` to be flags for
`go test`.
We then use environment variables to configure `gotestsum` (e.g.
`GOTESTSUM_JUNITFILE` is an env var accepted by `gotestsum`).

For cri tests, the test suite supports outputing test results to a
directory, these are in junit format already. The file is not named
properly just because the code that creates it (in ginkgo) is not
configured well. We can fix that upstream to give us a better name...
until then I'm keeping those results in a separate dir.

A second workflow is also added so the test results can be summed up and
a report added to the workflow run. The 2nd workflow is required for
this since PR runs do not have access to do some of this due to safety
reasons
(https://securitylab.github.com/research/github-actions-preventing-pwn-requests/)

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This commit is contained in:
Brian Goff
2021-03-25 19:11:29 +00:00
parent 1c05317a47
commit af1e2af720
5 changed files with 165 additions and 8 deletions

20
Vagrantfile vendored
View File

@@ -173,6 +173,18 @@ EOF
SHELL
end
config.vm.provision "install-gotestsum", type: "shell", run: "once" do |sh|
sh.upload_path = "/tmp/vagrant-install-gotestsum"
sh.inline = <<~SHELL
#!/usr/bin/env bash
source /etc/environment
source /etc/profile.d/sh.local
set -eux -o pipefail
${GOPATH}/src/github.com/containerd/containerd/script/setup/install-gotestsum
sudo cp ${GOPATH}/bin/gotestsum /usr/local/bin/
SHELL
end
# SELinux is Enforcing by default.
# To set SELinux as Disabled on a VM that has already been provisioned:
# SELINUX=Disabled vagrant up --provision-with=selinux
@@ -196,6 +208,8 @@ EOF
sh.upload_path = "/tmp/test-integration"
sh.env = {
'RUNC_FLAVOR': ENV['RUNC_FLAVOR'] || "runc",
'GOTEST': ENV['GOTEST'] || "go test",
'GOTESTSUM_JUNITFILE': ENV['GOTESTSUM_JUNITFILE'],
}
sh.inline = <<~SHELL
#!/usr/bin/env bash
@@ -213,6 +227,10 @@ EOF
#
config.vm.provision "test-cri", type: "shell", run: "never" do |sh|
sh.upload_path = "/tmp/test-cri"
sh.env = {
'GOTEST': ENV['GOTEST'] || "go test",
'REPORT_DIR': ENV['REPORT_DIR'],
}
sh.inline = <<~SHELL
#!/usr/bin/env bash
source /etc/environment
@@ -235,7 +253,7 @@ EOF
fi
trap cleanup EXIT
ctr version
critest --parallel=$(nproc) --ginkgo.skip='HostIpc is true'
critest --parallel=$(nproc) --report-dir="${REPORT_DIR}" --ginkgo.skip='HostIpc is true'
SHELL
end