Currently, if a crate wishes to run the coverage test with some cargo
features enabled, it must explicitly list them out in the
`crate_features` key of the coverage json files.
Change this so that if this key is omitted from the json file, then
test_coverage.py passes `--all-features` to cargo.
No functional change, as currently `crate_features` is a mandatory key,
so to opt-in to this new behavior, crates must remove this key from
their coverage json file(s).
Signed-off-by: Patrick Roy <roypat@amazon.co.uk>
It seems that the raw output of llvm-cov changed at some point, and we
ended upcomparing functoin coverage instead of line coverage. Fix this
by instead using the new json output, which will hopefully prevent such
goofs in the future.
Fixes#170
Signed-off-by: Patrick Roy <roypat@amazon.co.uk>
Before checking out different commits, clean the working directory to
avoid checkout failures to to temporary files created during benchmark
execution (as is done by linux-loader)
Signed-off-by: Patrick Roy <roypat@amazon.co.uk>
If we increase the coverage the message prints a negative value
as the difference from the previous value. This can be confusing
as it looks like the coverage has decreased.
For example if the value in coverage_config_x86_64.json is 73.42,
and the new coverage is 73.96 (increased), we have the following
error:
ValueError: Current code coverage (73.96%) deviates by -0.54%
from the previous code co...
Let's calculate the difference in reverse so that we have a
negative value if it decreases and positive otherwise.
Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
`black` suggests putting the string between single quotes so
there is no need to escape double quotes.
Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
We don't want the shell to interpret the contents of exclude_path,
but we want it all used as a single string for `--ignore-filename-regex`,
so let's put it between double quotes.
An example of failure is when we have | , as in this case:
$ CARGO_TARGET_DIR=./cov_build cargo llvm-cov test --summary-only \
--ignore-filename-regex crates/virtio-bindings|crates/virtio-queue/src/mock.rs \
--workspace --features=virtio-blk/backend-stdio
bash: crates/virtio-queue/src/mock.rs: Permission denied
...
error: failed to generate report: process didn't exit successfully: ...
(exit status: 74)
Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
cargo-kcov is broken on rust versions >=1.71.0. This is currently
blocking us from upgrading to the new toolchain version (1.72.0), which
is required to compile linux-loader (due to one dependency having a MSRV
policy of "N-2", meaning our current toolchain is no longer supported).
Signed-off-by: Patrick Roy <roypat@amazon.co.uk>
There are quite a few, genuine useful cases where one wants to overflow
the recommended 75 (some use 72) character rule.
Breaking long lines or manually breaking long compiler or other
tool-generated outputs is just confusing and error-prone.
This does not mean that we should stop wrapping manually written text
at a sane length, but enforcing the rule with a script seems to yield
questionable results while creating quite a bit of friction.
Signed-off-by: Erik Schilling <erik.schilling@linaro.org>
When the coverage changes, we raise exceptions with simple strings,
this causes the following error:
TypeError: exceptions must derive from BaseException
Let's use derivatives of BaseException to fix this issue.
Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
With newer versions of git we need to add an exception
for the `workdir` which is owned by a different user
then the one that is running the git commands.
Signed-off-by: Andreea Florescu <fandree@amazon.com>
The newer versions of git are checking the ownership of directories.
Since the `/workdir` is a shared directory with ownership of buildkite,
we need to add an exception for it.
Signed-off-by: Andreea Florescu <fandree@amazon.com>
Expand the title and changelog description lines to 60 and 75
respectively. That is in line with what other projects do and
allows for better narratives.
Signed-off-by: Mathieu Poirier <mathieu.poirier@linaro.org>
Instead of accounting for any difference in coverage, only fail the test
if the difference between the current coverage and the new coverage is
higher than 0.5%.
Signed-off-by: Andreea Florescu <fandree@amazon.com>
The test was assuming that we always want to run all the tests in an
workspace, but that is not the case when we have multiple crates that we
want to publish part of the same workspace.
To address this issue, we can now select to run the coverage test only
on the crate in scope. This can be achieved by passing the parameter
`--test-scope` with the value "crate". By default, tests are still
running as a workspace so that we do not need to update all the other
rust-vmm components that were counting on this feature.
The `test-scope` option can also be reused in other tests that are
making use of `--all`. For example, cargo build.
Signed-off-by: Andreea Florescu <fandree@amazon.com>
In order to be able to run the CI on rust-vmm-ci we needed to
make it a rust crate. Also the path to the coverage config
file needed to be updated in test_coverage.py so that the path
does not depend on the crate; a breadth-first search was used
to guarantee that the config file belongs to the crate that is
being tested.
Signed-off-by: Catalin Dumitru <catdum@amazon.com>
Set default values for the remote name and
base branch when the test is run locally and
the Buildkite variables are not available and
the variables BASE_BRANCH and REMOTE are not set.
Signed-off-by: Catalin Dumitru <catdum@amazon.com>
Although we check commit authors in test_commit_format.py to skip tests
for dependabot's commits, the current code only checks the author of the
head commit.
This PR make it check each commit's author so that we can test branches
where both a dependabot's commit and other commits exist.
Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
When dependabot updates the rust-vmm-ci submodule, the git commit
sumamry is over 50 chars. To be able to merge PR, exclude the dependabot
user from this checks.
Fixes: https://github.com/rust-vmm/rust-vmm-ci/issues/61
Signed-off-by: Andreea Florescu <fandree@amazon.com>
This commit adds a pytest command line option `--no-cleanup` for
the coverage test. Running the coverage test with this option will
let users save the `kcov_output` directory so that it is easier to
inspect the coverage report.
Signed-off-by: Henry Wang <henry.wang@arm.com>
Pytest closes stdin by default, so any processes
spawned from it won't have it. As some tests in
various rust-vmm crates might need stdin to be
open, we mock it with a (currently empty) byte
stream, which we can extend in the future.
Signed-off-by: Alexandra Iordache <aghecen@amazon.com>
- Added defines for PR branch, result files
- Updated variables & function names to be more explicit about what
they do
Signed-off-by: Andreea Florescu <fandree@amazon.com>
When there are no benchmarks on the master branch, instead of
crashing first check if the error. Pass the test in case
cargo bench failed due to main bench not being present.
Fixes: https://github.com/rust-vmm/rust-vmm-ci/issues/30
Signed-off-by: Andreea Florescu <fandree@amazon.com>
The test that was checking the commit message format
was not comparing the PR's HEAD to the right master,
even though `origin` was set to the pipeline repository.
Fixed this by fetching that repo. This way, `FETCH_HEAD`
points to the latest upstream master.
Signed-off-by: Laura Loghin <lauralg@amazon.com>
Added test that checks if commits follow the 50/72 git
commit rule and if they are signed (git commit -s).
Fixes: #9, #14.
Signed-off-by: Laura Loghin <lauralg@amazon.com>
1. Enabled the aarch64 coverage test in `test_coverage` script.
2. Seperated the coverage json sample file to two sample files for
different machine achitectures.
3. Updated Buildkite pipeline container version.
4. Updated README for the description of coverage files and container
version.
Signed-off-by: Henry Wang <henry.wang@arm.com>
This pipeline contains the tests that are currently run for
kvm-ioctls. The pipeline also has a step that runs the coverage
test defined in integration_tests/test_coverage.py.
The purpose of this pipeline is to be used by all rust-vmm
repositories to ensure we keep the same quality bar for all crates.
Signed-off-by: Andreea Florescu <fandree@amazon.com>
