Commit Graph

1 Commits

Author SHA1 Message Date
Patrick Ohly
535ab74346 e2e framework: track and report internal bugs
If something goes wrong during the test registration phase, the only solution
so far was to panic. This is not user-friendly and only allows to report one
problem at a time.

If initialization can continue, then a better solution is to record a bug,
continue, and then report all bugs together.

This also works when just listing tests. The new verify-e2e-suites.sh uses that
to check all test suites (identified as "packages that call
framework.AfterReadingAllFlags", with some exceptions) as part of
pull-kubernetes-verify.

Example output for a fake

    framework.RecordBug(framework.NewBug("fake bug during SIGDescribe", 0))

in test/e2e/storage/volume_metrics.go:
```
$ hack/verify-e2e-suites.sh
go version go1.21.1 linux/amd64
ERROR: E2E test suite invocation failed for test/e2e.
   ERROR: E2E suite initialization was faulty, these errors must be fixed:
   ERROR: test/e2e/storage/volume_metrics.go:49: fake bug during SIGDescribe
E2E suite test/e2e_kubeadm passed.
E2E suite test/e2e_node passed.
```
2023-10-10 18:15:46 +02:00