The YAML files get decoded into an unstructured object, without validation, and
then sent to the apiserver with a generic client. The default behavior is to
issue a warning to the client, which gets logged by client-go. What we want
instead is an error that causes the test to fail in a clean way right at the
beginning.
This makes "new" and "old" setup_env functions. In subsequent commits,
all callers of the "old" form will be fixed, and the "new" will be
renamed back.
The old and new functions diff:
```diff
--- /tmp/a 2023-12-14 09:02:57.804092696 -0800
+++ /tmp/b 2023-12-14 09:03:09.679999585 -0800
@@ -1,4 +1,4 @@
-kube::golang::old::setup_env() {
+kube::golang:🆕:setup_env() {
kube::golang::verify_go_version
# Set up GOPATH. We have tools which depend on being in a GOPATH (see
@@ -7,9 +7,9 @@
# Even in module mode, we need to set GOPATH for `go build` and `go install`
# to work. We build various tools (usually via `go install`) from a lot of
# scripts.
- # * We can't set GOBIN because that does not work on cross-compiles.
- # * We could use `go build -o <something>`, but it's subtle when it comes
- # to cross-compiles and whether the <something> is a file or a directory,
+ # * We can't just set GOBIN because that does not work on cross-compiles.
+ # * We could always use `go build -o <something>`, but it's subtle wrt
+ # cross-compiles and whether the <something> is a file or a directory,
# and EVERY caller has to get it *just* right.
# * We could leave GOPATH alone and let `go install` write binaries
# wherever the user's GOPATH says (or doesn't say).
@@ -20,16 +20,6 @@
#
# Eventually, when we no longer rely on run-in-gopath.sh we may be able to
# simplify this some.
- local go_pkg_dir="${KUBE_GOPATH}/src/${KUBE_GO_PACKAGE}"
- local go_pkg_basedir
- go_pkg_basedir=$(dirname "${go_pkg_dir}")
-
- mkdir -p "${go_pkg_basedir}"
-
- # TODO: This symlink should be relative.
- if [[ ! -e "${go_pkg_dir}" || "$(readlink "${go_pkg_dir}")" != "${KUBE_ROOT}" ]]; then
- ln -snf "${KUBE_ROOT}" "${go_pkg_dir}"
- fi
export GOPATH="${KUBE_GOPATH}"
# If these are not set, set them now. This ensures that any subsequent
@@ -40,24 +30,10 @@
# Make sure our own Go binaries are in PATH.
export PATH="${KUBE_GOPATH}/bin:${PATH}"
- # Change directories so that we are within the GOPATH. Some tools get really
- # upset if this is not true. We use a whole fake GOPATH here to collect the
- # resultant binaries.
- local subdir
- subdir=$(kube::realpath . | sed "s|${KUBE_ROOT}||")
- cd "${KUBE_GOPATH}/src/${KUBE_GO_PACKAGE}/${subdir}" || return 1
-
- # Set GOROOT so binaries that parse code can work properly.
- GOROOT=$(go env GOROOT)
- export GOROOT
-
# Unset GOBIN in case it already exists in the current session.
# Cross-compiles will not work with it set.
unset GOBIN
- # This seems to matter to some tools
- export GO15VENDOREXPERIMENT=1
-
- # Disable workspaces
- export GOWORK=off
+ # Explicitly turn on modules.
+ export GO111MODULE=on
}
```
Result: `make` works for k/k:
```
$ make kubectl
+++ [1211 11:07:31] Building go targets for linux/amd64
k8s.io/kubernetes/cmd/kubectl (static)
$ make WHAT=./cmd/kubectl/
+++ [1211 11:08:19] Building go targets for linux/amd64
k8s.io/kubernetes/./cmd/kubectl/ (non-static)
$ make WHAT=k8s.io/kubernetes/cmd/kubectl
+++ [1211 11:08:52] Building go targets for linux/amd64
k8s.io/kubernetes/cmd/kubectl (static)
```
Result: `make` works for staging by package:
```
$ make WHAT=k8s.io/api
+++ [1211 11:11:37] Building go targets for linux/amd64
k8s.io/api (non-static)
```
Result: `make` fails for staging by path:
```
$ make WHAT=./staging/src/k8s.io/api
+++ [1211 11:12:44] Building go targets for linux/amd64
k8s.io/kubernetes/./staging/src/k8s.io/api (non-static)
cannot find module providing package k8s.io/kubernetes/staging/src/k8s.io/api: import lookup disabled by -mod=vendor
(Go version in go.work is at least 1.14 and vendor directory exists.)
!!! [1211 11:12:44] Call tree:
!!! [1211 11:12:44] 1: /home/thockin/src/kubernetes/hack/lib/golang.sh:850 kube::golang::build_some_binaries(...)
!!! [1211 11:12:44] 2: /home/thockin/src/kubernetes/hack/lib/golang.sh:1012 kube::golang::build_binaries_for_platform(...)
!!! [1211 11:12:44] 3: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
!!! [1211 11:12:44] Call tree:
!!! [1211 11:12:44] 1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
!!! [1211 11:12:44] Call tree:
!!! [1211 11:12:44] 1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
make: *** [Makefile:96: all] Error 1
```
Result: `make test` fails:
```
$ make test WHAT=./cmd/kubectl
+++ [1211 11:13:38] Set GOMAXPROCS automatically to 6
+++ [1211 11:13:38] Running tests without code coverage and with -race
cmd/kubectl/kubectl.go:25:2: cannot find package "k8s.io/client-go/plugin/pkg/client/auth" in any of:
/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/plugin/pkg/client/auth (vendor tree)
/home/thockin/sdk/gotip/src/k8s.io/client-go/plugin/pkg/client/auth (from $GOROOT)
/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/client-go/plugin/pkg/client/auth (from $GOPATH)
cmd/kubectl/kubectl.go:20:2: cannot find package "k8s.io/component-base/cli" in any of:
/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/component-base/cli (vendor tree)
/home/thockin/sdk/gotip/src/k8s.io/component-base/cli (from $GOROOT)
/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/component-base/cli (from $GOPATH)
cmd/kubectl/kubectl.go:21:2: cannot find package "k8s.io/kubectl/pkg/cmd" in any of:
/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd (vendor tree)
/home/thockin/sdk/gotip/src/k8s.io/kubectl/pkg/cmd (from $GOROOT)
/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubectl/pkg/cmd (from $GOPATH)
cmd/kubectl/kubectl.go:22:2: cannot find package "k8s.io/kubectl/pkg/cmd/util" in any of:
/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util (vendor tree)
/home/thockin/sdk/gotip/src/k8s.io/kubectl/pkg/cmd/util (from $GOROOT)
/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubectl/pkg/cmd/util (from $GOPATH)
make: *** [Makefile:191: test] Error 1
```
Previously, SELinuxMount started two pods and in laboratory conditions
waited for the second Pod to get stuck (because of RWOP) and observed
kubelet behavor after the test unstuck them (i.e. deleted the first Pod).
When testing RWO volumes, the second Pod may not get stuck, it may actually
run. So update the tests to allow the second Pod to run and start counting
CSI calls for it earlier.
very simple tests to check that the node controllers removes the taint.
Test will be used later for add more coverage without having to depend
on e2e tests for validating complex scenarios and combinations of flags
and controllers.
Change-Id: I55ce01488352cce9d673585e7959678c2bfb5924
- Increase the global level for broadcaster's logging to 3 so that users can ignore event messages by lowering the logging level. It reduces information noise.
- Making sure the context is properly injected into the broadcaster, this will allow the -v flag value to be used also in that broadcaster, rather than the above global value.
- test: use cancellation from ktesting
- golangci-hints: checked error return value
Extending the duration and the allowed delta in f6682370b1 was still not enough
to make the unit test run reliably in pull-kubernetes-unit.
Now it uses the original, stricter timing again, but only when run locally. In
Prow (detected by checking the "CI" env variable), the duration check is
skipped.
The dead code was found with:
deadcode -test -filter=k8s.io/kubernetes/test/e2e/framework/... ./test/e2e ./test/e2e_node ./test/e2e_node ./test/e2e_kubeadm
See https://go.dev/blog/deadcode for an introduction.
This removes everything that was reported as dead code for
test/e2e/framework/providers.
* Forensic Container Checkpointing as described in KEP 2008 moves from
Alpha to Beta. This is corresponding code change.
* Adapt e2e test to handle
'(rpc error: code = Unimplemented desc = unknown method CheckpointContainer'
and
'(rpc error: code = Unimplemented desc = method CheckpointContainer not implemented)'
and
'(rpc error: code = Unknown desc = checkpoint/restore support not available)'
One error message is if the CRI implementation does
not implement the CRI RPC (too old) and the second is
if the CRI implementation does explicitly not support the feature.
The third error message can be seen if the container engine
explicitly disabled the checkpoint/restore support,
* As described in the corresponding KEP 2008 explicitly test for
disabled functionality.
* Extended test to look for the checkpoint kubelet metric.
* Extended test to look for the CRI error metric.
* Add separate sub-resource permission to control permissions on
the checkpoint kubelet API endpoint
Signed-off-by: Adrian Reber <areber@redhat.com>
When failing inside the `ginkgo.By` callback function, skipping intermediate
stack frames didn't work properly because `ginkgo.By` itself and other internal
code is also on the stack.
To fix this, the code which can fail now runs outside of such a
callback. That's not a big loss, the only advantage of the callback was getting
timing statistics from Ginkgo which weren't used in practice.
Since v2.45, the `stress` subcommand was added and the CI issue was fixed:
- kubernetes/kubernetes PR 123258
- kubernetes/kubernetes PR 123284
- kubernetes/k8s.io PR 6422
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
When stopping polling, the provided messages becomes the complete failure
message. This means that the code which calls gomega.StopTrying must include
the pod in the message instead of just summarizing the phase. This makes the
failure more useful.
agnhost was updated to v2.46 in PR 123258, but the image was not
published due to the failue of `post-kubernetes-push-e2e-agnhost-test-images`
(issue 123266).
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
Fix issue 123266
> CI: `post-kubernetes-push-e2e-agnhost-test-images` is failing
> (`gcr.io/k8s-staging-e2e-test-images/agnhost:2.46-linux-amd64 is a manifest list`)
To avoid creating a manifest list with the recent version of buildx,
`--provenance=false --sbom=false` has to be specified.
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
Merge vishh/stress@eab4e3384b into
agnhost.
Old usage: `stress -mem-alloc-size 12Mi -mem-alloc-sleep 10s -mem-total 4Gi`
New usage: `agnhost stress --mem-alloc-size 12Mi --mem-alloc-sleep 10s --mem-total 4Gi`
This is a part of the steps to migrate from legacy Schema 1 images
(issue 123146)
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
Adding a doc comment for all existing items makes it more obvious that new
items should be documented more carefully. It also has the welcome side effect
that each item gets indented independently from the others.
With a dynamic client and a rest mapper it is possible to load arbitrary YAML
files and create the object defined by it. This is simpler than adding specific
Go code for each supported type.
Because the version now matters, the incorrect version in the DRA YAMLs were
found and fixed.
ktesting.TContext combines several different interfaces. This makes the code
simpler because less parameters need to be passed around.
An intentional side effect is that the apiextensions client interface becomes
available, which makes it possible to use CRDs. This will be needed for future
DRA tests.
Support for CRDs depends on starting the apiserver via
k8s.io/kubernetes/cmd/kube-apiserver/app/testing because only that enables the
CRD extensions. As discussed on Slack, the long-term goal is to replace the
in-tree StartTestServer with the one in staging, so this is going in the right
direction.
The new TContext interface combines a normal context and the testing interface,
then adds some helper methods. The context gets canceled when the test is done,
but that can also be requested earlier via Cancel.
The intended usage is to pass a single `tCtx ktesting.TContext` parameter
around in all helper functions that get called by a unit or integration test.
Logging is also more useful: Log[f] and Fatal[f] output is prefixed with
"[FATAL] ERROR: " to make it stand out more from regular log output.
If this approach turns out to be useful, it could be extended further (for
example, with a per-test timeout) and might get moved to a staging repository
to enable usage of it in other staging repositories.
To allow other implementations besides testing.T and testing.B, a custom
ktesting.TB interface gets defined with the methods expected from the
actual implementation. One such implementation can be ginkgo.GinkgoT().
Currently published image version doesn't include latest code changes,
for example CDI support.
Manifests include different and outdated image versions
snuck in there while I was working on the test, but is ultimately not necessary to test the functionality.
skipping healthz check resulted in leaking goroutines from poststarthooks.
Don't implement interfaces that trigger tests with in-line and
pre-provisioned vSphere volumes.
With cloud provider removal, the in-tree vSphere tests won't be able to
create a volume in vSphere and thus test in-line volumes in Pods and
pre-provisioned PVs. Only dynamically provisioned volumes can be used for
testing, because they're provisioned by the vSphere CSI driver.
Refactor the code related to creating an internal type load balancer in the e2e tests for network load balancers. The modification removes the check for the "azure" provider and updates it to only check for "gke" and "gce" providers. This change ensures that the test only runs when the cluster is using "gke" or "gce" as the provider. The counterpart test is in the out-of-tree cloud provider azure.
This moves adding a pod to ReservedFor out of the main scheduling cycle into
PreBind. There it is done concurrently in different goroutines. For claims
which were specifically allocated for a pod (the most common case), that
usually makes no difference because the claim is already reserved.
It starts to matter when that pod then cannot be scheduled for other reasons,
because then the claim gets unreserved to allow deallocating it. It also
matters for claims that are created separately and then get used multiple times
by different pods.
Because multiple pods might get added to the same claim rapidly independently
from each other, it makes sense to do all claim status updates via patching:
then it is no longer necessary to have an up-to-date copy of the claim because
the patch operation will succeed if (and only if) the patched claim is valid.
Server-side-apply cannot be used for this because a client always has to send
the full list of all entries that it wants to be set, i.e. it cannot add one
entry unless it knows the full list.
* put var in local
Signed-off-by: husharp <jinhao.hu@pingcap.com>
* revert gomod
Signed-off-by: husharp <jinhao.hu@pingcap.com>
---------
Signed-off-by: husharp <jinhao.hu@pingcap.com>
ginkgo.GinkgoHelper is a recent addition to ginkgo which allows functions to
mark themselves as helper. This then changes which callstack gets reported for
failures. It makes sense to support the same mechanism also for logging.
There's also no reason why framework.Logf should produce output that is in a
different format than klog log entries. Having time stamps formatted
differently makes it hard to read test output which uses a mixture of both.
Another user-visible advantage is that the error log entry from
framework.ExpectNoError now references the test source code.
With textlogger there is a simple replacement for klog that can be reconfigured
to let the caller handle stack unwinding. klog itself doesn't support that
and should be modified to support it (feature freeze).
Emitting printf-style output via that logger would work, but become less
readable because the message string would get quoted instead of printing it
verbatim as before. So instead, the traditional klog header gets reproduced
in the framework code. In this example, the first line is from klog, the second
from Logf:
I0111 11:00:54.088957 332873 factory.go:193] Registered Plugin "containerd"
...
I0111 11:00:54.987534 332873 util.go:506] >>> kubeConfig: /var/run/kubernetes/admin.kubeconfig
Indention is a bit different because the initial output is printed before
installing the logger which writes through ginkgo.GinkgoWriter.
One welcome side effect is that now "go vet" detects mismatched parameters for
framework.Logf because fmt.Sprintf is called without mangling the format
string. Some of the calls were incorrect.
A stand-alone binary shouldn't import the test/e2e/framework, which is targeted
towards usage in a Ginkgo test suite. This currently works, but will break once
test/e2e/framework becomes more opinionated about how to configure logging.
The simplest solution is to duplicate the one short function that the binary
was calling in the framework.
Now that we have it (8a89a1f5a5), let's also make sure that
the new WithFlaky is used everywhere instead if [Flaky]. This way it can be
used for filtering by label.
Using klog.Fatal to abort a test leads to a poor user experience because the
output is buffered in ginkgo.GinkgoWriter and not flushed before killing the
process. The output is also different from other failures. Using the normal
error checking is better.
Before:
$ KUBECONFIG=/no/such/config go test -v ./test/e2e/
Jan 19 10:06:58.475: INFO: The --provider flag is not set. Continuing as if --provider=skeleton had been used.
=== RUN TestE2E
I0119 10:06:58.475844 99472 e2e.go:109] Starting e2e run "5303f626-ae0e-44d7-abf1-b4956d910ef4" on Ginkgo node 1
Running Suite: Kubernetes e2e suite - /nvme/gopath/src/k8s.io/kubernetes/test/e2e
=================================================================================
Random Seed: 1705655217 - will randomize all specs
Will run 4678 of 7421 specs
goroutine 817 [running]:
k8s.io/klog/v2/internal/dbg.Stacks(0x0)
/nvme/gopath/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/internal/dbg/dbg.go:35 +0x85
k8s.io/klog/v2.(*loggingT).output(0x9d92b20, 0x3, 0x0, 0xc00069d7a0, 0x2, {0x834c6e8?, 0x9d91c80?}, 0x300000060?, 0x0)
...
k8s.io/klog/v2.Fatal(...)
/nvme/gopath/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1652
k8s.io/kubernetes/test/e2e.setupSuite({0x7fb49064c078, 0xc003072360})
/nvme/gopath/src/k8s.io/kubernetes/test/e2e/e2e.go:187 +0x125
...
FAIL k8s.io/kubernetes/test/e2e 0.759s
FAIL
After:
$ KUBECONFIG=/no/such/config go test -v ./test/e2e/
Jan 19 10:12:58.889: INFO: The --provider flag is not set. Continuing as if --provider=skeleton had been used.
=== RUN TestE2E
I0119 10:12:58.889224 106019 e2e.go:109] Starting e2e run "bed5a77a-f595-42d0-b512-5f601067444b" on Ginkgo node 1
Running Suite: Kubernetes e2e suite - /nvme/gopath/src/k8s.io/kubernetes/test/e2e
=================================================================================
Random Seed: 1705655578 - will randomize all specs
Will run 4678 of 7421 specs
------------------------------
[SynchronizedBeforeSuite] [FAILED] [0.001 seconds]
[SynchronizedBeforeSuite]
/nvme/gopath/src/k8s.io/kubernetes/test/e2e/e2e.go:69
Timeline >>
Jan 19 10:12:59.063: INFO: >>> kubeConfig: /no/such/config
Jan 19 10:12:59.063: INFO: Unexpected error: Error loading client:
<*errors.errorString | 0xc00182c130>:
error creating client: error loading KubeConfig: open /no/such/config: no such file or directory
{
s: "error creating client: error loading KubeConfig: open /no/such/config: no such file or directory",
}
[FAILED] in [SynchronizedBeforeSuite] - /nvme/gopath/src/k8s.io/kubernetes/test/e2e/e2e.go:186 @ 01/19/24 10:12:59.064
<< Timeline
[FAILED] Error loading client: error creating client: error loading KubeConfig: open /no/such/config: no such file or directory
In [SynchronizedBeforeSuite] at: /nvme/gopath/src/k8s.io/kubernetes/test/e2e/e2e.go:186 @ 01/19/24 10:12:59.064
------------------------------
Summarizing 1 Failure:
[FAIL] [SynchronizedBeforeSuite]
/nvme/gopath/src/k8s.io/kubernetes/test/e2e/e2e.go:186
Ran 0 of 7421 Specs in 0.001 seconds
FAIL! -- A BeforeSuite node failed so all tests were skipped.
--- FAIL: TestE2E (0.18s)
FAIL
FAIL k8s.io/kubernetes/test/e2e 0.769s
FAIL
This replaces the klog formatting and message routing with a simpler
implementation that uses less code. The main difference is that we skip the
entire unused message routing.
Instead, the same split output streams as for JSON gets implemented in the
io.Writer implementation that gets passed to the textlogger.
The dead code was found with:
deadcode -test -filter=k8s.io/kubernetes/test/e2e/framework/... ./test/e2e ./test/e2e_node ./test/e2e_node ./test/e2e_kubeadm
See https://go.dev/blog/deadcode for an introduction.
Only dead code which is clearly not needed anymore (glog logging),
questionable (skipping based on feature gates) or
redundant (WaitForPodSuccessInNamespaceSlow) gets removed for now. More
removals might make sense in the future.
