Commit Graph

99 Commits

Author SHA1 Message Date
Erik Schilling
7c1057e9bc test_commit_format: no longer check body length
There are quite a few, genuine useful cases where one wants to overflow
the recommended 75 (some use 72) character rule.

Breaking long lines or manually breaking long compiler or other
tool-generated outputs is just confusing and error-prone.

This does not mean that we should stop wrapping manually written text
at a sane length, but enforcing the rule with a script seems to yield
questionable results while creating quite a bit of friction.

Signed-off-by: Erik Schilling <erik.schilling@linaro.org>
2023-07-14 11:42:08 +01:00
Stefano Garzarella
9dfe5b267c Use derivatives of BaseException for exceptions
When the coverage changes, we raise exceptions with simple strings,
this causes the following error:

    TypeError: exceptions must derive from BaseException

Let's use derivatives of BaseException to fix this issue.

Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
2023-06-28 12:44:28 +01:00
Jonathan Woollett-Light
7e9af57588 test: Update code coverage test
Signed-off-by: Jonathan Woollett-Light <jcawl@amazon.co.uk>
2023-06-16 11:46:25 +01:00
Jonathan Woollett-Light
56bfdc97bb test: Format with black
Signed-off-by: Jonathan Woollett-Light <jcawl@amazon.co.uk>
2023-06-16 11:46:25 +01:00
Patrick Roy
285971e8c7 Update container version to 24
v24 contains the rust-src component which miri requires.

Signed-off-by: Patrick Roy <roypat@amazon.co.uk>
2023-05-23 12:01:47 +02:00
Patrick Roy
589b6b1414 Update CODEOWNERS
Adding @roypat and @JonathanWoollett-Light from the firecracker team.

Signed-off-by: Patrick Roy <roypat@amazon.co.uk>
2023-05-22 17:22:26 +00:00
Patrick Roy
0beb66692d Update container version to v23
This version contains an updated rust toolchain (1.68.2 -> 1.69.0), as
well as the miri component for the nightly toolchain.

Signed-off-by: Patrick Roy <roypat@amazon.co.uk>
2023-05-22 11:40:10 +03:00
Patrick Roy
8627b3766b Update container to v22
This revision contains critcmp and cargo-audit on ARM again, after the
previous revision was lacking them due to an OOM at container build
time.

Closes #123

Signed-off-by: Patrick Roy <roypat@amazon.co.uk>
2023-04-13 13:40:51 +02:00
Viresh Kumar
3640704b02 Cargo audit is missing for arm platforms
Lets run it only for x86 for the moment.

Signed-off-by: Viresh Kumar <viresh.kumar@linaro.org>
2023-04-05 10:17:12 +02:00
Viresh Kumar
7ffe41abf0 update to container v21
That updates to a newer version of Rust, 1.68.2, and adds back
libclang-dev package which is required by vhost-device/gpio crates.

An earlier version, v20, is skipped here as we missed updating
rust-vmm-ci along with it. v20 updated documentation and triggered
builds for changes in build_container.sh.

Signed-off-by: Viresh Kumar <viresh.kumar@linaro.org>
2023-04-05 10:17:12 +02:00
Andreea Florescu
3f9869f285 update to container v19
There should be no changes required in the crates. The only
difference is in the libgpio library, as we're using a release
and not a certain commit on the main branch.

This is the first container built with the automation, and in
the process we had to do some significant cleanup to remove
things that were not needed. Hopefully this will not trigger
any errors in the CIs.

Signed-off-by: Andreea Florescu <fandree@amazon.com>
2023-03-28 18:07:47 +08:00
Andreea Florescu
c2f8c93e37 update container version to v18
This container version updates the Rust version to 1.67.1, the base
Ubuntu image to 22.04 (which is the newest that still works with
kcov), and adds the gpio library needed by the vhost-device tests.

Signed-off-by: Andreea Florescu <fandree@amazon.com>
2023-02-13 12:37:31 +02:00
Andreea Florescu
607c775dde update container to v17
This new version pulls the latest available Rust version (1.66.1)

Signed-off-by: Andreea Florescu <fandree@amazon.com>
2023-01-11 14:40:37 +02:00
Andreea Florescu
aa33f19d6b temporarily increase the timeout to 15 mins
We need to increase the timeout for each test to 15 minutes because we
are experiencing timeouts when running the tests. I think this might be
actually related to the hosts running the CI, because some early
investigations by aghecen@amazon.com show that the threads are blocked
on doing epoll_wait. This is odly similar to the issue discussed here
related to epoll: https://lore.kernel.org/all/Y1pY2n6E1Xa58MXv@kroah.com/

For more details, check: https://github.com/rust-vmm/community/issues/137

Signed-off-by: Andreea Florescu <fandree@amazon.com>
2022-11-30 17:38:29 +02:00
Andreea Florescu
45443cc21e add hypervisor tag for agents
By default we are now using the kvm hypervisor in tests.
This can be overritten for the entire pipeline at the pipeline
level by using the environment variable `AGENT_TAG_HYPERVISOR`.

Fixes: https://github.com/rust-vmm/rust-vmm-ci/issues/115

Signed-off-by: Andreea Florescu <fandree@amazon.com>
2022-11-30 17:38:29 +02:00
Andreea Florescu
e8c8fc32b0 update docker plugin
We were using a rather outdated version. Moving to the latest available
one.

Signed-off-by: Andreea Florescu <fandree@amazon.com>
2022-11-30 17:38:29 +02:00
Muminul Islam
4cb208e71e .buildkite: add hypervisor to special_keys
Without this fix, yml is generated wrongly like below:

steps:
- label: coverage-x86_64
  command: pytest $(find . -type f -name "test_coverage.py")
  retry:
    automatic: false
  agents:
    os: linux
    platform: x86_64.metal
    hypervisor: mshv
  plugins:
  - docker#v3.8.0:
      image: rustvmm/dev:v16
      always-pull: true
      privileged: true
  timeout_in_minutes: 5
  hypervisor: mshv // This is now right

Signed-off-by: Muminul Islam <muislam@microsoft.com>
2022-11-24 14:38:23 +08:00
Muminul Islam
3e1f6fc125 autogenerate_pipeline: Add hypervisor tag
In order to support Microsoft Hypervisor, this tag is used
to identify agents that has MSHV support

Signed-off-by: Muminul Islam <muislam@microsoft.com>
2022-11-18 11:48:08 +02:00
Andreea Florescu
7af2e16d91 add support for specifying the agent queue
This is needed so we can have separate queues for tests that require
special instances. For example, running performance tests require
nothing else to run at the same time.

Signed-off-by: Andreea Florescu <fandree@amazon.com>
2022-10-26 11:14:59 +03:00
Andreea Florescu
e2e2918a73 generate pipeline: support for FW unknown keys
Any keys in the input JSON that we do not recognize we are adding
as they are to the output pipeline. This is needed so that we don't
have to always update the autogeneration script for each new Buildkite
configuration that we want to support.

Signed-off-by: Andreea Florescu <fandree@amazon.com>
2022-10-26 11:14:59 +03:00
Andreea Florescu
b0edd172fc autogenerate_pipeline: use dictionary instead of self
In the script for autogenerating the rust-vmm-ci tests, we were using
self directly for setting the configuration of the tests, and then
we were returning vars(self). This was a pretty obfuscated way of
returning a JSON, so instead we're now just explicitly declaring
a dictionary as part of self, and returning that instead. This
is needed for future extensions as well as we want to be able to
extend the tool such that it supports any keys from the input
to be passed as they are so that we don't need to keep changing it
whenever we need to add support for another Buildkite configuration
parameter.

Signed-off-by: Andreea Florescu <fandree@amazon.com>
2022-10-26 11:14:59 +03:00
Andreea Florescu
edd221ac3b don't allow undocumented unsafe blocks
We already have this as a requirement for merging PRs as part of the
PR checklist template, but it's better to enforce it. This comes with
the disadvantage that we will need to update all comments and add the
prefix "SAFETY: " as required by the clippy check.

Signed-off-by: Andreea Florescu <fandree@amazon.com>
2022-10-25 10:53:24 +03:00
Andreea Florescu
5f36cc9604 fix whitespace problems in autogenerate script
Signed-off-by: Andreea Florescu <fandree@amazon.com>
2022-09-12 13:05:35 +03:00
Andreea Florescu
2eab2752c0 allow the timeout to be overwritten in input
Before this patch the timeout could only be overwritten in the
TIMEOUT environment variable. While that approach works and it
is still needed so we can set a different timeout depending
on the repository on which we run the pipeline, we also need a
way to easily set a different timeout in custom tests. This
comes handy when we have custom pipeline for certain crates
and we want to be able to easily specify a timeout in a
more human readable way.

Signed-off-by: Andreea Florescu <fandree@amazon.com>
2022-09-12 13:05:35 +03:00
Andreea Florescu
1401badaaf fix test benchmark
With newer versions of git we need to add an exception
for the `workdir` which is owned by a different user
then the one that is running the git commands.

Signed-off-by: Andreea Florescu <fandree@amazon.com>
2022-09-12 12:20:15 +03:00
Andreea Florescu
258161e88a update README with latest version
Also added a note that the version in the readme is not necessarily
the latest one.

Signed-off-by: Andreea Florescu <fandree@amazon.com>
2022-08-26 19:48:52 +03:00
Andreea Florescu
6fbd7c7355 fix test commit format
The newer versions of git are checking the ownership of directories.
Since the `/workdir` is a shared directory with ownership of buildkite,
we need to add an exception for it.

Signed-off-by: Andreea Florescu <fandree@amazon.com>
2022-08-26 19:48:52 +03:00
Andreea Florescu
b3ed1c9253 update to container version 16
This version adds some tooling for fuzzing and updated to Rust 1.63.0.

Signed-off-by: Andreea Florescu <fandree@amazon.com>
2022-08-26 19:48:52 +03:00
Mathieu Poirier
99fe2eb2e0 Relax 50/72 rule for more descriptive titles
Expand the title and changelog description lines to 60 and 75
respectively.  That is in line with what other projects do and
allows for better narratives.

Signed-off-by: Mathieu Poirier <mathieu.poirier@linaro.org>
2022-02-22 10:06:54 +02:00
Andreea Florescu
ee6be8ca18 make coverage test less sensitive to changes
Instead of accounting for any difference in coverage, only fail the test
if the difference between the current coverage and the new coverage is
higher than 0.5%.

Signed-off-by: Andreea Florescu <fandree@amazon.com>
2022-02-14 10:36:49 +01:00
Sergii Glushchenko
d216a46879 Update to rustvmm/dev:v15 with Rust 1.58.1
Signed-off-by: Sergii Glushchenko <gsserge@amazon.com>
2022-01-21 17:04:21 +02:00
Sergey Glushchenko
d023262164 Use v14 container image with Rust 1.54
Signed-off-by: Sergii Glushchenko <gsserge@amazon.com>
2022-01-18 17:21:11 +02:00
Laura Loghin
b6858b232d run cargo audit with --deny warnings enabled
It is better to fail on warnings as well when running `cargo
audit` so we can catch problems in time.
Fixes https://github.com/rust-vmm/rust-vmm-ci/issues/52.

Signed-off-by: Laura Loghin <lauralg@amazon.com>
2022-01-11 15:11:28 +01:00
Sergey Glushchenko
aee82cf0a4 Add @gsserge to CODEOWNERS
Signed-off-by: Sergii Glushchenko <gsserge@amazon.com>
2021-11-01 16:50:08 +01:00
Andreea Florescu
b037be3396 update test_coverage to allow running it on crates
The test was assuming that we always want to run all the tests in an
workspace, but that is not the case when we have multiple crates that we
want to publish part of the same workspace.

To address this issue, we can now select to run the coverage test only
on the crate in scope. This can be achieved by passing the parameter
`--test-scope` with the value "crate". By default, tests are still
running as a workspace so that we do not need to update all the other
rust-vmm components that were counting on this feature.

The `test-scope` option can also be reused in other tests that are
making use of `--all`. For example, cargo build.

Signed-off-by: Andreea Florescu <fandree@amazon.com>
2021-10-28 14:14:38 +03:00
Ahmed Abouzied
7931077cdc Assert no spaces in crate features JSON
Spaces in the `crate_features` JSON value can cause errors when running
kcov.

Fixes: https://github.com/rust-vmm/rust-vmm-ci/issues/74

Signed-off-by: Ahmed Abouzied <email@aabouzied.com>
2021-10-11 16:40:34 +03:00
Sergii Glushchenko
68d4dbf85c Check formatting of code examples in doc comments
Add format_code_in_doc_comments=true to the cargo fmt
config options.

Fixes: https://github.com/rust-vmm/rust-vmm-ci/issues/58

Signed-off-by: Sergii Glushchenko <gsserge@amazon.com>
2021-10-06 17:39:17 +03:00
Andreea Florescu
7f22582590 update container version to v13
In the new container version we're using ubuntu 20.04 as the base image
instead of ubuntu 18.04.

Signed-off-by: Andreea Florescu <fandree@amazon.com>
2021-09-24 14:20:45 +03:00
alindima
ce13bfb1ef Make build warnings fatal, again
Add the RUSTFLAGS="-D warnings" to cargo build
commands

The yaml config file has been deleted in the meantime,
so we need to add this again.

Signed-off-by: alindima <alindima@amazon.com>
2021-09-23 19:19:35 +03:00
Catalin Dumitru
472dea07fb Document default timeout for pipeline step
Mentioned that each step in the autogenerated pipeline has a
default timeout of 5 minutes.

Signed-off-by: Catalin Dumitru <catdum@amazon.com>
2021-09-23 17:11:18 +03:00
Catalin Dumitru
f67ef4c84d Update README.md
Included the new environment variable `TIMEOUTS_MIN` for overriding
step timeouts and updated the "Custom Pipeline" section to
reflect that we are also autogenerating the custom pipelines.

Signed-off-by: Catalin Dumitru <catdum@amazon.com>
2021-09-16 17:11:28 +03:00
Catalin Dumitru
dd754bb88e Add timeout for Buildkite steps
Added a default timeout value for each Buildkite step that can be
overridden through the environment variable `TIMEOUTS_MIN`.

Signed-off-by: Catalin Dumitru <catdum@amazon.com>
2021-09-16 17:11:28 +03:00
Catalin Dumitru
e58c6f698f Remove static pipeline
The use of the static file pipeline.yml is now deprecated. All
pipelines should use the dynamic script autogenerate_pipeline.py.

Signed-off-by: Catalin Dumitru <catdum@amazon.com>
2021-09-16 16:34:19 +03:00
alindima
6e1bdaac8e Make build warnings fatal
Add the RUSTFLAGS="-D warnings" to cargo build
commands

Signed-off-by: alindima <alindima@amazon.com>
2021-09-15 10:24:01 +03:00
Catalin Dumitru
ae7db2d98a Rename master to main
Updated occurrences of "master" with "main".

Signed-off-by: Catalin Dumitru <catdum@amazon.com>
2021-09-02 12:33:44 +03:00
Catalin Dumitru
1311bfa03f Update clippy test command
Added the option --all-targets to the command in the clippy test for
clippy to also run on the integration tests.

Signed-off-by: Catalin Dumitru <catdum@amazon.com>
2021-09-01 15:59:39 +03:00
Catalin Dumitru
d7ca3dc9f2 Add help option for scripts
Added help options -h, --help for the script that autogenerates
the pipeline and for the one that runs the tests locally. In the
former, replaced the module optparse (which is now deprecated)
with the module argparse.

Signed-off-by: Catalin Dumitru <catdum@amazon.com>
2021-09-01 14:29:22 +03:00
Catalin Dumitru
8901e77522 Leftovers from #79
Added copyright info and fixed typos.

Signed-off-by: Catalin Dumitru <catdum@amazon.com>
2021-08-26 12:34:31 +03:00
Catalin Dumitru
555474aaba Add script to run tests locally
The script uses the test_description.json file to get the test
names and the commands to be run.

Signed-off-by: Catalin Dumitru <catdum@amazon.com>
2021-08-26 12:34:31 +03:00
Catalin Dumitru
cd3d97f923 Allow skipping tests when generating the pipeline
Added a new environment variable for skipping tests if needed.

Signed-off-by: Catalin Dumitru <catdum@amazon.com>
2021-08-25 16:25:48 +03:00