The existing walk.go and conformance.txt have a few shortcomings
which we'd like to resolve:
- difficult to get the full test name due to test context nesting
- complicated AST logic and understanding necessary due to the
different ways a test can be invoked and written
This changes the AST parsing logic to be much more simple and simply
looks for the comments at/around a specific line. This file/line
information (and the full test name) is gathered by a custom ginkgo
reporter which dumps the SpecSummary data to a file.
Also, the SpecSummary dump can, itself, be potentially useful for
other post-processing and debugging tasks.
Signed-off-by: John Schnake <jschnake@vmware.com>
This also revealed that the regex for Context was too generous and would
catch things like SecurityContext or ContextWithFoo(...) calls
This ensures that test suites with a ineligible tag in their top-level
Describe will be rejected from promotion to conformance
If there are tags in the test name that describe qualities of the
test that make it ineligible for conformance, raise an error. This
is basically the "skip list" that heptio's e2e image used to use.
Thankfully all of our existing Conformance tests lack these tags. I
considered added [Slow] to the list, but let's save that for another
day.
To detect whether conformance tests have been changed, the test
argument with the `ConformanceIt` is recorded to conformance.txt.
This change remove additional tags (e.g., [Feature:]) and trim spaces,
so that the detection is less-prone to noise.
This change also updates conformance.txt with normalized names.
This test creates a golden list of existing conformance tests. Efforts
to add or remove conformance tests will require you to rebuild the
golden list, and changes to the golden list will be reviewed by
sig-architecture.
