Ginkgo v1 had a much longer default test timeout, in v2 this
switched to being 1 hour. This is not long enough to run many of our
suites.
Here we copy the backwards compatibility that is used by
hack/gingo-e2e.sh to unbreak serial pipelines.
Ginkgo has been migrated to V2, add this to unwanted dependencies
so that it won't be shown up as a dep again in the future.
Signed-off-by: Dave Chen <dave.chen@arm.com>
The alias for vendor/github.com/onsi/ginkgo/ginkgo ensures that code like
30e99cb2a9/experiment/kind-conformance-image-e2e.sh (L110)
continues to work. The one without "vendor/" is there just in case that it
was used because it also worked.
Long term, "ginkgo" is a nicer, version independent alias. It gets used
internally to avoid future churn and gets documented also publicly in the
Makefile help.
The caveat is that there's no guarantee that a future v3 CLI will be compatible
with current invocations. But the most common usage is through
hack/ginkgo-e2e.sh, which can deal with such differences.
Default timeout setting has been reduced from `24h` down to `1h` in
Ginkgo V2, but for some long running test this is too short.
How long to abort the test was controlled by the the linux command `timeout`
in V1. e.g. `'timeout -k 30s 150m ...`, and is configured in the file
like `sig-network-misc.yaml`.
Set the timeout manually for Ginkgo V2 to avoid the early aborting.
Signed-off-by: Dave Chen <dave.chen@arm.com>
The change is needed for `verify-e2e-test-ownership.sh`.
The `jq` is re-defined since the structure of test spec
is different with v1 and the stacktrace related validation
is not available, e.g. `package` and `func`.
Signed-off-by: Dave Chen <dave.chen@arm.com>
The test/e2e directory contains several unit tests that should run as part of
"make test":
./test/e2e/chaosmonkey/chaosmonkey_test.go
./test/e2e/storage/external/external_test.go
./test/e2e/storage/utils/utils_test.go
./test/e2e/framework/log_test.go
./test/e2e/framework/testfiles/testfiles_test.go
./test/e2e/framework/timer/timer_test.go
./test/e2e/framework/node/wait_test.go
./test/e2e/framework/pod/resource_test.go
./test/e2e/framework/config/config_test.go
./test/e2e/framework/ingress/ingress_utils_test.go
./test/e2e/framework/providers/gce/firewall_test.go
Because they were excluded by "./test/e2e/*", some of them became outdated.
./test/e2e/e2e_test.go is the only test that needs to be excluded because it is
the E2E test suite that depends on a functional cluster.
The following investigation occurred during development.
Add TimingHistogram impl that shares lock with WeightedHistogram
Benchmarking and profiling shows that two layers of locking is
noticeably more expensive than one.
After adding this new alternative, I now get the following benchmark
results.
```
(base) mspreitz@mjs12 kubernetes % go test -benchmem -run=^$ -bench ^BenchmarkTimingHistogram$ k8s.io/component-base/metrics/prometheusextension
goos: darwin
goarch: amd64
pkg: k8s.io/component-base/metrics/prometheusextension
cpu: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
BenchmarkTimingHistogram-16 22232037 52.79 ns/op 0 B/op 0 allocs/op
PASS
ok k8s.io/component-base/metrics/prometheusextension 1.404s
(base) mspreitz@mjs12 kubernetes % go test -benchmem -run=^$ -bench ^BenchmarkTimingHistogram$ k8s.io/component-base/metrics/prometheusextension
goos: darwin
goarch: amd64
pkg: k8s.io/component-base/metrics/prometheusextension
cpu: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
BenchmarkTimingHistogram-16 22190997 54.50 ns/op 0 B/op 0 allocs/op
PASS
ok k8s.io/component-base/metrics/prometheusextension 1.435s
```
and
```
(base) mspreitz@mjs12 kubernetes % go test -benchmem -run=^$ -bench ^BenchmarkTimingHistogramDirect$ k8s.io/component-base/metrics/prometheusextension
goos: darwin
goarch: amd64
pkg: k8s.io/component-base/metrics/prometheusextension
cpu: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
BenchmarkTimingHistogramDirect-16 28863244 40.99 ns/op 0 B/op 0 allocs/op
PASS
ok k8s.io/component-base/metrics/prometheusextension 1.890s
(base) mspreitz@mjs12 kubernetes %
(base) mspreitz@mjs12 kubernetes %
(base) mspreitz@mjs12 kubernetes % go test -benchmem -run=^$ -bench ^BenchmarkTimingHistogramDirect$ k8s.io/component-base/metrics/prometheusextension
goos: darwin
goarch: amd64
pkg: k8s.io/component-base/metrics/prometheusextension
cpu: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
BenchmarkTimingHistogramDirect-16 27994173 40.37 ns/op 0 B/op 0 allocs/op
PASS
ok k8s.io/component-base/metrics/prometheusextension 1.384s
```
So the new implementation is roughly 20% faster than the original.
Add overlooked exception, rename timingHistogram to timingHistogramLayered
Use the direct (one mutex) style of TimingHistogram impl
This is about a 20% gain in CPU speed on my development machine, in
benchmarks without lock contention. Following are two consecutive
trials.
(base) mspreitz@mjs12 prometheusextension % go test -benchmem -run=^$ -bench Histogram .
goos: darwin
goarch: amd64
pkg: k8s.io/component-base/metrics/prometheusextension
cpu: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
BenchmarkTimingHistogramLayered-16 21650905 51.91 ns/op 0 B/op 0 allocs/op
BenchmarkTimingHistogramDirect-16 29876860 39.33 ns/op 0 B/op 0 allocs/op
BenchmarkWeightedHistogram-16 49227044 24.13 ns/op 0 B/op 0 allocs/op
BenchmarkHistogram-16 41063907 28.82 ns/op 0 B/op 0 allocs/op
PASS
ok k8s.io/component-base/metrics/prometheusextension 5.432s
(base) mspreitz@mjs12 prometheusextension % go test -benchmem -run=^$ -bench Histogram .
goos: darwin
goarch: amd64
pkg: k8s.io/component-base/metrics/prometheusextension
cpu: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
BenchmarkTimingHistogramLayered-16 22483816 51.72 ns/op 0 B/op 0 allocs/op
BenchmarkTimingHistogramDirect-16 29697291 39.39 ns/op 0 B/op 0 allocs/op
BenchmarkWeightedHistogram-16 48919845 24.03 ns/op 0 B/op 0 allocs/op
BenchmarkHistogram-16 41153044 29.26 ns/op 0 B/op 0 allocs/op
PASS
ok k8s.io/component-base/metrics/prometheusextension 5.044s
Remove layered implementation of TimingHistogram
This commit cleans up references to the old kubernetes-node-e2e-images
project. In the process it removes the `LIST_IMAGES` mode as listing
large numbers of public cloud projects is not particularly useful, and
has been somewhat broken for a long period of time - as we defaulted
launching a VM to a different project than listing.
This commit undoes the GODEBUG=x509sha1=1 workaround.
The problem should be fixed in Go 1.18.1 now.
Signed-off-by: Madhav Jivrajani <madhav.jiv@gmail.com>
* Introduce networking/v1alpha1 api, ClusterCIDRConfig type
Introduce networking/v1alpha1 api group.
Add `ClusterCIDRConfig` type to networking/v1alpha1 api group, this type
will enable the NodeIPAM controller to support multiple ClusterCIDRs.
* Change ClusterCIDRConfig.NodeSelector type in api
* Fix review comments for API
* Update ClusterCIDRConfig API Spec
Introduce PerNodeHostBits field, remove PerNodeMaskSize
Over time the size of our junit xml has exploded to the point where
test-grid fails to process them. We still have the original/full
*.stdout files from where the junit xml files are generated from so the
junit xml files need NOT have the fill/exact output for
processing/display. So let us prune the large messages with an
indicator that we have "[... clipped...]" some of the content so folks
can see that they have to consult the full *.stdout files.
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
Running logcheck as part of golangci-lint has several advantages:
- faster checking because finding files and parsing is shared
with other linters
- gets rid of the complex and buggy
hack/verify-structured-logging.sh (https://github.com/kubernetes/kubernetes/issues/106746)
- support for // nolint:logcheck
- works with Go 1.18
This should fix the following error when running
./hack/update-generated-stable-metrics.sh:
'go get' is no longer supported outside a module.
To build and install a command, use 'go install' with a version,
like 'go install example.com/cmd@latest'
For more information, see https://golang.org/doc/go-get-install-deprecation
or run 'go help get' or 'go help install'.
Using `go get` to download gopkg.in/yaml.v2 package into
KUBE_EXTRA_GOPATH directory no longer works. Interestingly, main repo
already has gopkg.in/yaml.v2@v2.4.0, same version that was installed by
that go get.
I guess that GOPATH with multiple elements no longer works either,
and since this code was the only user of KUBE_EXTRA_GOPATH, let's remove
it as well.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
- Also update test-cmd.sh to pass a signing ca to the kube controller
manager, so CSRs work properly in integration tests.
Signed-off-by: Margo Crawford <margaretc@vmware.com>
This has somewhat subtle implications. For a concrete example, this
changes the `-trimpath` behavior from only affecting the named pkg to
affecting all pkgs, which broke ginkgo, which seems to try to strip its
own `pwd` prefix. But since that runs in run-in-gopath, and not in
KUBE_ROOT, it fails to strip anything.
e.g.
before this, strings in the binary would be like
/home/user/kube/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/...
Ginkgo would find its own root as
/home/user/kube/_output/local/go/src/k8s.io/kubernetes/
so it would produce
vendor/github.com/onsi/ginkgo/...
in logs.
after this, strings in the binary strip the KUBE_ROOT and be like:
_output/local/go/src/k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/...
Ginkgo would find its own root as
/home/user/kube/_output/local/go/src/k8s.io/kubernetes/
so it would not strip anything, and produce
_output/local/go/src/k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/...
in logs.
before:
```
$ make generated_files
+++ [0226 13:42:17] Building go targets for linux/amd64:
hack/make-rules/helpers/go2make
> non-static build: k8s.io/kubernetes/hack/make-rules/helpers/go2make
```
after:
```
$ make generated_files
+++ [0226 14:30:08] Building go targets for linux/amd64
k8s.io/kubernetes/hack/make-rules/helpers/go2make (non-static)
```
When running an integration test that measures performance, like for example
test/integration/scheduler_perf, running etcd with debug level output is
undesirable because it creates additional load on the system and isn't
realistic.
The default is still "debug", but ETCD_LOGLEVEL=warn can be used to override
that.
Since removing dockershim, `make test-e2e-node` will fail by default as
there is no provided container runtime endpoint.
This commit defaults us to using containerd's default socket path as the
local test target, rather than failing hard.
When CRDs are deleted, discovery local cache is not invalidated.
This brings about `resource not found` error when new CRD with same name is created
with different fields(ie. changing scope from cluster-wide to namespaced).
Because this already deleted CRD still stays in serverresources.json and kubectl tries to use it.
This local cached files have 10 minutes TTL. After deletion, if user waits 10 minutes,
files will be expired and deleted and there will be no errors. However, 10 minutes is a long time
and cache needs to be invalidated after deletion occurs.
This PR adds a document into delete command by noting that there might be a need to invalidate discovery
cache when CRD is deleted. In addition to that this PR adds a test to catch this behavior.
This currently covers two cases:
- "kubectl list" (the regression from https://github.com/kubernetes/kubernetes/issues/107012)
- "kubectl get pods/no-such-pod" (no particular reason except that the output
should be deterministic)
In contrast to some other tests that check for strings inside the
output (run_deprecated_api_tests) or compare after
sorting (run_kubectl_version_tests), stdout, stderr and the return code must
match exactly.
This ensures that there is no extra, unexpected output and that the right
output stream is used.
Right now, `run_remote.go` only supports GCE instances. But actually
running the tests is completely independent of GCE and could work just
as well on any SSH-accessible machine.
This patch adds a new `--mode` switch, which defaults to `gce` for
backwards compatibility, but can be set to `ssh`. In that mode, the GCE
API is not used at all, and we simply connect to the hosts given via
`--hosts`.
This is still better than `run_local.go` because the latter mixes build
environment with test environment, which doesn't fit well with
container-optimized operating systems.
This is part of an effort to setup the e2e node tests on Fedora CoreOS
(see https://github.com/coreos/fedora-coreos-tracker/issues/990).
Patch best viewed with whitespace ignored.
Minor. This removes a few "testdata" dirs from .make/all_go_dirs.mk
-hack/make-rules/helpers/go2make/testdata/dir-with-gofiles
-test/conformance/testdata
-test/instrumentation/testdata/pkg/kubelet/metrics
-test/instrumentation/testdata/staging/src/k8s.io/metrics
-test/typecheck/testdata/bad
-test/typecheck/testdata/good
-test/typecheck/testdata/good/testdata
-vendor/k8s.io/kubectl/pkg/cmd/edit/testdata
This list of dirs is used to figure out which directories need codegen,
and none of these do.
Signed-off-by: wangyysde <net_use@bzhy.com>
Generation swagger.json.
Use v2 path for hpa_cpu_field.
run update-codegen.sh
Signed-off-by: wangyysde <net_use@bzhy.com>
Some tests have a short timeout for starting the pods (1 minute), but if
those tests happen to be the first ones to run, and the images have to be
pulled, then the test could timeout, especially with larger images. This
commit will allow us to prepull commonly used E2E test images, so this issue
can be avoided.
The --log-file parameter will be deprecated as of Kubernetes 1.23 and should be
avoided. The replacement for distroless images is the image with go-runner, a
tool that handles output redirection.
For kubemark to run in that image it must be built as static binary.
kube-log-runner was formerly known as go-runner when it was originally introduced
in 393e0952e9
It was moved to kubernetes/release/images/build/go-runner later but is now
needed again in Kubernetes itself as replacement for the deprecated --log-file
klog feature: when bringing up a Windows node, kube-proxy.exe and kubelet.exe must be wrapped
with the helper binary to redirect output.
It got renamed to avoid a naming conflict with
test/conformance/image/go-runner and because the name was too vague.
Other downstream Kubernetes users may have a similar need, therefore it makes
sense to provide a prebuilt binary also in the release archives.
With Kubernetes 1.19.X no longer supported, there are no supported branches using bazel. Developers should not have errant bazel files, this will be left to reviewers to catch.
This commit forces Kubelet Configuration files to always be generated
and when possible will use the kubeletconfig file that has been provided
by the test orchestrator
Resolves https://github.com/kubernetes/kubernetes/issues/102975
shellcheck errors are printed to stdout by default, hence they need to be redirected
to stderr in order to be well parsed for Junit representation by `juLog` function.
hack/verify-structured-logging.sh now tests migrated packages
for use of unstructured logging functions and all other packages
for use of correct structured logging patterns.
Signed-off-by: Umanga Chapagain <chapagainumanga@gmail.com>
On a ppc64le host, fast build was failing:
# make KUBE_FASTBUILD=true quick-release
+++ [0908 15:56:36] Verifying Prerequisites....
+++ [0908 15:56:36] Building Docker image kube-build:build-fd009aaa81-5-v1.23.0-go1.17-buster.0
+++ [0908 15:56:44] Syncing sources to container
+++ [0908 15:56:48] Running build command...
+++ [0908 15:56:56] Building go targets for linux/ppc64le:
./vendor/k8s.io/code-generator/cmd/prerelease-lifecycle-gen
> non-static build: k8s.io/kubernetes/./vendor/k8s.io/code-generator/cmd/prerelease-lifecycle-gen
touch: cannot touch '_output/bin/prerelease-lifecycle-gen': No such file or directory
make[2]: *** [Makefile.generated_files:209: _output/bin/prerelease-lifecycle-gen] Error 1
make[1]: *** [Makefile:552: generated_files] Error 2
make: *** [Makefile:512: cross] Error 1
!!! [0908 15:56:58] Call tree:
!!! [0908 15:56:58] 1: build/../build/common.sh:476 kube::build::run_build_command_ex(...)
!!! [0908 15:56:58] 2: build/release.sh:36 kube::build::run_build_command(...)
make: *** [Makefile:454: quick-release] Error 1
This error happened because the _output/bin/ directory didn't exist at
the moment the file _output/bin/prerelease-lifecycle-gen was touched, so
the path didn't exist.
The _output/bin symlink was not created by kube::golang::place_bins()
because kube::golang::setup_platforms() assumed
KUBE_CLIENT_PLATFORMS=linux/amd64 despite being on a ppc64le host.
Fix build error by not assuming host_arch=amd64 when on ppc64le in
kube::golang::setup_platforms().
Signed-off-by: Murilo Opsfelder Araujo <muriloo@linux.ibm.com>
This addresses a problem that occurred when trying the following replacement:
hack/pin-dependency.sh k8s.io/klog/v2=github.com/pohly/klog/v2 flush-log-sink
That itself worked, but update-vendor.sh then failed:
=== tidying go.mod/go.sum in staging/src/k8s.io/code-generator/examples
go: k8s.io/apimachinery@v0.0.0 requires
k8s.io/klog/v2@v2.4.1-0.20210909144920-0b91caa8c854: invalid version: unknown revision 0b91caa8c854
go: k8s.io/apimachinery@v0.0.0 requires
k8s.io/klog/v2@v2.4.1-0.20210909144920-0b91caa8c854: invalid version: unknown revision 0b91caa8c854