go list -find takes ~60% the time:
$ time go list -e ./... | grep -E -v "/(build|third_party|vendor|staging|clientset_generated|hack)/" | md5sum
b5593b3f51f3b3cd08c33bbff9627d10 -
real 0m2.687s
user 0m3.624s
sys 0m1.552s
$ time go list -find -e ./... | grep -E -v "/(build|third_party|vendor|staging|clientset_generated|hack)/" | md5sum
b5593b3f51f3b3cd08c33bbff9627d10 -
real 0m1.721s
user 0m1.675s
sys 0m1.197s
https://github.com/kubernetes/kubernetes/pull/116166#discussion_r1123924871
To run just a specific linter based on command line flags, the default
configuration needs to be disabled (because it would enable additional ones)
and then command line flags must be passed through to "golangci-lint run".
For example, to lint with just "go vet" in verbose mode, use:
verify-golangci-lint.sh -c none -- --disable-all --enable=govet -v
LC_ALL is always wanted and GREP_OPTIONS is never wanted. The `grep
--color=never` dates back to 2016, an issue with OLD grep on Macs, which
was hard to deal with when this was all Makefile magic. Now that it's a
script, we can do it simpler.
Because the script now explicitly selects the configuration file, the files no
longer have to be in the root directory. Having them in hack without the
leading dot is better because they then have the same owners as the script and
are more visible.
The downside is that manual invocations of golangci-lint without the parameter
no longer work.
All wrappers except for ExpectNoError are identical to their gomega
counterparts. The only advantage that they have is that their invocations are
shorter.
That advantage does not outweigh their disadvantages:
- cannot be used in combination with gomega.Eventually/Consistently
- not a full replacement for gomega, so we just end up using both
- don't support passing a stack offset and thus cannot be used in helper
functions
- ginkgolinter does not work for them, so sub-optimal calls like this one
are not reported:
framework.ExpectEqual(len(items), 0)
->
gomega.Expect(items).To(gomega.BeEmpty())
- developers try to make do with what's available in the framework, leading
to sub-optimal checks like this:
framework.ExpectEqual(true, strings.Contains(event.Message, expectedEventError), "Event error should indicate non-root policy caused container to not start")
->
gomega.Expect(event.Message).To(gomega.ContainSubstring(expectedEventError), "Event error should indicate non-root policy caused container to not start")
So let's remove these wrappers. As a first step they get marked as deprecated.
This enables stricter
linting (https://github.com/kubernetes/kubernetes/pull/109728), once enabled,
to report new code which uses them.
we should only use this env var for `arm`, since `arm64` is fully
supported by etcd folks, let us drop this!
(ex - https://github.com/etcd-io/etcd/releases/tag/v3.5.6)
ppc64le comment should be dropped as well
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
Several files that I maintain (primarily logcheck.conf right now, but
I'm also working on golangci-lint) are in this directory. It would be
useful to have approval rights to avoid delays.
The previous approach was based on the observation that some Prow jobs use the
--report-dir parameter instead of the E2E_REPORT_DIR env variable. Parsing the
command line was necessary to use the --json-report and --junit-report
parameters.
But that is complex and can be avoided by triggering the creation of complete
reports in the E2E test suite. The paths are hard-coded and relative to the
report directory to keep the code simple.
There was a report that k8s-triage started processing more data after
6db4b741dd was merged. It's unclear whether
that was because of the new <report-dir>/ginkgo_report.xml file. To avoid
this potential problem, the reports are now in a "ginkgo" sub-directory.
While at it, error checking gets enhanced:
- Create directories at the start of
the suite and bail out early if that fails.
- *All* e2e suites using the framework do this, not just test/e2e.
- Added missing error checking of truncated JUnit report writing.
Bump golangci-lint version. This version
adds support for go1.20.
This consequently also bumps the version of
staticcheck. Note that this was changed to
use master to be compatible with 1.20. This
commit reverts back to using a tagged release.
Signed-off-by: Madhav Jivrajani <madhav.jiv@gmail.com>
Swagger "docs" are actually Go code, which is used by other codegen
tools, so if you really want to regen EVERYTHING, this is part of it and
sequence matters.
Calling update-codegen.sh with no arguments runs all the functions in
definition order. Client-generation depends on protobuf, so protobuf
codegen needs to be near the beginning.
Also add some debug output for protobuf generation.
Also hide some old, verbose debug output.
Now update-generated-proto-bindings rules all the api.pb.go generation.
Running this shows no delta on the runtime.pb.go
This exposes an issue in how protoc is called for protos that specify
`go_package` which is fixed here.
Not all of our protos specify that option (even though it is
recommended), which will be fixed subsequently.
Each of these scripts is basically identical, and all were too brittle.
Now they should be more resilient and easier to manage. The script
still needs to be updated if we add new ones, which I do not love.
More cleanup to follow.
The `find` tool has hard to comprehend syntax and does not consider
things excluded by .gitignore. I keep tripping over this in my own
repos, where I have __stuff which gets found.
This converts update-codegen to use `git ls-files` in a seemingly
equivalent way (`-cmo --exclude-standard`). I verified it finds the
same set of files as before.
This also drops some obsolete filtering.
Also hide grep errors for not-found files, which can happen if a file is
removed but git ls-files still knows it.
Re-running update-codegen shows no diffs.
This will make subsequent changes easier.
Don't just grep for DO NOT EDIT - anchor it in something that looks like
a comment and alone on a line.
Also ignore __* dirs
Prevent it from triggering on update-generated-swagger-docs (hack, but
better than before)
The env vars are needed until go workspaces lands, then it can get
simpler.
Downsides to this:
1) If you don't call kube::golang::setup_env, it might work but will
just splat results somewhere
2) The resultant binaries are not in _output/bin but instead in the
phony GOPATH/bin (which setup_env puts in PATH)
hack/pin-dependency.sh github.com/moby/ipvs v1.1.0
- go to a fixed tag for `vishvananda/netns`
- no more references to `pkg/errors`
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
This came up when updating go-oidc. After updating go-oidc (with its
dependency tree), cloud.google.com/go was no longer used as a package
import, but still listed in the module dependency graph; as a result,
"go mod vendor" no longer pulled in cloud.google.com/go itself, but
update-vendor-licenses.sh still wanted a license file for it since it
appeared in the list of modules.
This scenario is already supposed to be handled: when a module doesn't
contain any *files* as first-level content, if the number of
subdirectories it contains *equals* the number of submodules it
contains (excluding itself), the module is skipped. This fails for
cloud.google.com/go because several submodules are included in the
module dependency graph but aren't actually used, and therefore not
vendored.
Updating the test to check that the number of subdirectories is less
than or equal to the number of expected submodules fixes this.
The correct fix would be to process the submodules first, keeping a
note of which ones really have content, then check that the top-level
module only contains subdirectories corresponding to those modules;
but it's not clear to me that this is worth the effort (especially in
a shell script).
Signed-off-by: Stephen Kitt <skitt@redhat.com>
From the warning message that ginkgo now emits:
--slow-spec-threshold is deprecated --slow-spec-threshold has been deprecated
and will be removed in a future version of Ginkgo. This feature has proved
to be more noisy than useful. You can use --poll-progress-after, instead, to
get more actionable feedback about potentially slow specs and understand
where they might be getting stuck.
We already use --poll-progress-after.
This is a fix for 104aab81a4: because
the default was not set for E2E_TEST_DEBUG_TOOL, all parameters were always
also passed to the E2E suite.
That wasn't wrong for the parameters so far, but breaks when using something
like --output-dir which is only understood by the CLI.
If the script was called with no arguments, it passed "${@:-}" to the suite,
which expands to one empty argument. That's not right, "${@}" should be used
instead because it expands to nothing when empty.
Every ginkgo callback should return immediately when a timeout occurs or the
test run manually gets aborted with CTRL-C. To do that, they must take a ctx
parameter and pass it through to all code which might block.
This is a first automated step towards that: the additional parameter got added
with
sed -i 's/\(framework.ConformanceIt\|ginkgo.It\)\(.*\)func() {$/\1\2func(ctx context.Context) {/' \
$(git grep -l -e framework.ConformanceIt -e ginkgo.It )
$GOPATH/bin/goimports -w $(git status | grep modified: | sed -e 's/.* //')
log_test.go was left unchanged.
The kubectl-convert binary itself only depends on the following runtime
libraries when linking dynamically:
```
> ldd _output/bin/kubectl-convert
linux-vdso.so.1 (0x00007ffef0786000)
libpthread.so.0 => /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/libpthread.so.0 (0x00007f5f4ac25000)
libdl.so.2 => /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/libdl.so.2 (0x00007f5f4ac20000)
libc.so.6 => /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/libc.so.6 (0x00007f5f4aa00000)
/nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/ld-linux-x86-64.so.2 => /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib64/ld-linux-x86-64.so.2 (0x00007f5f4ac2c000)
```
We now move kubectl-convert to become a static binary as well to achieve
maximum portability.
Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
The kube-aggregator itself only depends on the following runtime
libraries when linking dynamically:
```
> ldd _output/bin/kube-aggregator
linux-vdso.so.1 (0x00007fff1616f000)
libpthread.so.0 => /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/libpthread.so.0 (0x00007fad9339a000)
libc.so.6 => /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/libc.so.6 (0x00007fad93000000)
/nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/ld-linux-x86-64.so.2 => /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib64/ld-linux-x86-64.so.2 (0x00007fad933a1000)
```
We now move the kube-aggregator to become a static binary as well to
achieve maximum portability.
Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
We need to better catch problems like the one we fixed in:
7b6a3d6a33
Essentially a copy-paste error where had a branch pointing to staging/src/k8s.io/code-generator instead of staging/src/k8s.io/apimachinery
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
The call to `i18n.LoadTranslations` needs to occur on init so
that package-level variables that call `i18n.T()` to initialize
their values will be able to receive the translated string.
Added new integration tests to test help output translation.
github.com/container-orchestrated-devices/container-device-interface and
github.com/opencontainers/runtime-tools depend on some problematic packages in
code that isn't used by Kubernetes. Explicitly listing those dependencies
ensures that they do not get pulled in accidentally.
Dependencies need to be updated to use
github.com/container-orchestrated-devices/container-device-interface.
It's not decided yet whether we will implement Topology support
for DRA or not. Not having any toppology-related code
will help to avoid wrong impression that DRA is used as a hint
provider for the Topology Manager.
This adds a new resource.k8s.io API group with v1alpha1 as version. It contains
four new types: resource.ResourceClaim, resource.ResourceClass, resource.ResourceClaimTemplate, and
resource.PodScheduling.
This was originally added in the early days of go.mod in order to ensure that the root k8s.io/kubernetes
go.mod file reflected all the versions of dependencies we were using, and to ensure that go commands didn't spuriously
increment these versions.
With module pruning in go1.17+ and go defaulting to mod=readonly mode in go1.16+, this is no longer necessary.
Most parameters can be passed to both the CLI and the suite, but some
(for example, --ginkgo.slow-spec-threshold) had no effect when only
passed to the suite.
DOCKER is otherwise used to be the command name (perhaps podman), but we were conflating DOCKER_OPTS in kube::util::ensure_docker_daemon_connectivity.
Split out docker opts.
This fixes shellcheck warning that docker is assigned an array and then a string in some scripts.
Adding the ability to ignore no schedule flags in testing.
Specifically node.cloudprovider.kubernetes.io/uninitialized:NoSchedule
Fix shellcheck complaint.
The `make` rules which auto-generate some of our API stuff are
incredibly baroque, and hard to maintain. They were originally added on
the assumption that we would stop checking generated files into git.
Since then we have moved away from that goal, and the worst problems
with generated files have been resolved.
Reasons to kill this:
* It is slow on every build, as opposed to just being slow when running
the generators. It is even slow to calculate that there's nothing to
update.
* Most development work doesn't involve changing APIs.
* It only covers about half (or less) of the generated code, and making
it cover more would be even slower.
* Approximately 1 person knows how this all works.
* We have CI to make sure changes do not get merged without updating
this code.
* We have corner cases where this does the WRONG thing and tracking
those down is ugly and hard in perpetuity.
So this commit puts all the same logic that WAS in the
Makefile.generated_files into update-codegen.sh.
I do not love this script, especially WRT sub-packages, but I am trying
not to boil the ocean. I hope to follow up with some more cleanups over
time.
I have tested this manually and with the scripts and it still seems to
catch errors properly.
This includes a change to kube::util::read-array to make it not unset
variables and not over-write non-array variables.
A staging repo which just got created with only the doc.go file in it won't
have any dependencies yet, which caused the script to fail because the
dependency files didn't get created:
+++ [0926 14:33:22] go.mod: tidying
cat: /tmp/update-vendor.1VTv/group_replace.ZbIT/go.mod.require_direct.tmp: No such file or directory
!!! [0926 14:33:23] Call tree:
!!! [0926 14:33:23] 1: hack/update-vendor.sh:354 group_directives(...)
- Moves kms proto apis to the staging repo
- Updates generate and verify kms proto scripts to check staging repo
Signed-off-by: Anish Ramasekar <anish.ramasekar@gmail.com>
The new support in ginkgo for progress reports while a test runs dumps
information about where a test is stuck when it runs too long. This can provide
additional insights into what the test is waiting for.
For the Kubernetes jobs using ginkgo-e2e.sh, such dumps are now enabled after
300 seconds and then get repeated every 20 seconds. The initial delay is
intentionally the same as for warning about a slow test. The rationale is that
such test runtimes are unexpected and may need further information to diagnose
why they are slow.
With -ginkgo.source-root, Ginkgo is able to locate the Kubernetes source code
and display small source code snippets for functions that are related to the
test, determined through a heuristic that assumes that all files under the test
suite are for the tests in it.
Set intercept mode to none will help to reveal more information when the
test hangs,
- https://github.com/onsi/ginkgo/issues/970
or circumvent cases where the code grabbing the stdout/stderr pipe
is not under the framework control and may cause hangs,
- https://github.com/onsi/ginkgo/issues/851
The flag `output-interceptor-mode` is set to `none` as we were trying to
figure out of the rootcase of the test flaky, it's only intended for debugging.
- https://github.com/kubernetes/kubernetes/issues/111086
But this set also has some side effect, since it will turn off stdout/stderr
capture completely, any output to stdout/stderr will be lost.
Now that the root cause is not caused by Ginkgo bump nor how the intercept
mode was set, we'd better to follow the default value.
Signed-off-by: Dave Chen <dave.chen@arm.com>
Some scripts and tools still relied on the deprecated flags, the ones
which are about to be removed.
This is intentionally not a complete removal of all those flags in the entire
repo. This would lead to much more code churn also in places where commands
still accept the flags because they use klog directly.
Introduce networking/v1alpha1 api group.
Add `ClusterCIDR` type to networking/v1alpha1 api group, this type
will enable the NodeIPAM controller to support multiple ClusterCIDRs.
- add feature gate
- add encrypted object and run generated_files
- generate protobuf for encrypted object and add unit tests
- move parse endpoint to util and refactor
- refactor interface and remove unused interceptor
- add protobuf generate to update-generated-kms.sh
- add integration tests
- add defaulting for apiVersion in kmsConfiguration
- handle v1/v2 and default in encryption config parsing
- move metrics to own pkg and reuse for v2
- use Marshal and Unmarshal instead of serializer
- add context for all service methods
- check version and keyid for healthz
Signed-off-by: Anish Ramasekar <anish.ramasekar@gmail.com>
This change is to promote local storage capacity isolation feature to GA
At the same time, to allow rootless system disable this feature due to
unable to get root fs, this change introduced a new kubelet config
"localStorageCapacityIsolation". By default it is set to true. For
rootless systems, they can set this configuration to false to disable
the feature. Once it is set, user cannot set ephemeral-storage
request/limit because capacity and allocatable will not be set.
Change-Id: I48a52e737c6a09e9131454db6ad31247b56c000a
This applies to all jobs using hack/ginkgo-e2e.sh. This is done because
Spyglass does not render the escape sequences, making test output harder to
read.
It is done here because then we don't need to set GINKGO_NO_COLOR in all the
different Prow job configs.
