Currently, if a crate wishes to run the coverage test with some cargo
features enabled, it must explicitly list them out in the
`crate_features` key of the coverage json files.
Change this so that if this key is omitted from the json file, then
test_coverage.py passes `--all-features` to cargo.
No functional change, as currently `crate_features` is a mandatory key,
so to opt-in to this new behavior, crates must remove this key from
their coverage json file(s).
Signed-off-by: Patrick Roy <roypat@amazon.co.uk>
It seems that the raw output of llvm-cov changed at some point, and we
ended upcomparing functoin coverage instead of line coverage. Fix this
by instead using the new json output, which will hopefully prevent such
goofs in the future.
Fixes#170
Signed-off-by: Patrick Roy <roypat@amazon.co.uk>
If we increase the coverage the message prints a negative value
as the difference from the previous value. This can be confusing
as it looks like the coverage has decreased.
For example if the value in coverage_config_x86_64.json is 73.42,
and the new coverage is 73.96 (increased), we have the following
error:
ValueError: Current code coverage (73.96%) deviates by -0.54%
from the previous code co...
Let's calculate the difference in reverse so that we have a
negative value if it decreases and positive otherwise.
Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
`black` suggests putting the string between single quotes so
there is no need to escape double quotes.
Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
We don't want the shell to interpret the contents of exclude_path,
but we want it all used as a single string for `--ignore-filename-regex`,
so let's put it between double quotes.
An example of failure is when we have | , as in this case:
$ CARGO_TARGET_DIR=./cov_build cargo llvm-cov test --summary-only \
--ignore-filename-regex crates/virtio-bindings|crates/virtio-queue/src/mock.rs \
--workspace --features=virtio-blk/backend-stdio
bash: crates/virtio-queue/src/mock.rs: Permission denied
...
error: failed to generate report: process didn't exit successfully: ...
(exit status: 74)
Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
cargo-kcov is broken on rust versions >=1.71.0. This is currently
blocking us from upgrading to the new toolchain version (1.72.0), which
is required to compile linux-loader (due to one dependency having a MSRV
policy of "N-2", meaning our current toolchain is no longer supported).
Signed-off-by: Patrick Roy <roypat@amazon.co.uk>
When the coverage changes, we raise exceptions with simple strings,
this causes the following error:
TypeError: exceptions must derive from BaseException
Let's use derivatives of BaseException to fix this issue.
Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
Instead of accounting for any difference in coverage, only fail the test
if the difference between the current coverage and the new coverage is
higher than 0.5%.
Signed-off-by: Andreea Florescu <fandree@amazon.com>
The test was assuming that we always want to run all the tests in an
workspace, but that is not the case when we have multiple crates that we
want to publish part of the same workspace.
To address this issue, we can now select to run the coverage test only
on the crate in scope. This can be achieved by passing the parameter
`--test-scope` with the value "crate". By default, tests are still
running as a workspace so that we do not need to update all the other
rust-vmm components that were counting on this feature.
The `test-scope` option can also be reused in other tests that are
making use of `--all`. For example, cargo build.
Signed-off-by: Andreea Florescu <fandree@amazon.com>
In order to be able to run the CI on rust-vmm-ci we needed to
make it a rust crate. Also the path to the coverage config
file needed to be updated in test_coverage.py so that the path
does not depend on the crate; a breadth-first search was used
to guarantee that the config file belongs to the crate that is
being tested.
Signed-off-by: Catalin Dumitru <catdum@amazon.com>
This commit adds a pytest command line option `--no-cleanup` for
the coverage test. Running the coverage test with this option will
let users save the `kcov_output` directory so that it is easier to
inspect the coverage report.
Signed-off-by: Henry Wang <henry.wang@arm.com>
Pytest closes stdin by default, so any processes
spawned from it won't have it. As some tests in
various rust-vmm crates might need stdin to be
open, we mock it with a (currently empty) byte
stream, which we can extend in the future.
Signed-off-by: Alexandra Iordache <aghecen@amazon.com>
Added test that checks if commits follow the 50/72 git
commit rule and if they are signed (git commit -s).
Fixes: #9, #14.
Signed-off-by: Laura Loghin <lauralg@amazon.com>
1. Enabled the aarch64 coverage test in `test_coverage` script.
2. Seperated the coverage json sample file to two sample files for
different machine achitectures.
3. Updated Buildkite pipeline container version.
4. Updated README for the description of coverage files and container
version.
Signed-off-by: Henry Wang <henry.wang@arm.com>
This pipeline contains the tests that are currently run for
kvm-ioctls. The pipeline also has a step that runs the coverage
test defined in integration_tests/test_coverage.py.
The purpose of this pipeline is to be used by all rust-vmm
repositories to ensure we keep the same quality bar for all crates.
Signed-off-by: Andreea Florescu <fandree@amazon.com>
