This involves moving the report files, but it allows me to delete the
indirect variable and indirect array code in update-codgen. As proud as
I was of figuring that out, I am also ashamed of myself for doing it.
This is my atonement.
Use the "subprojects" aspect of update-codegen to generat openapi for
the subprojects. Next we can simplify and remove the generic support.
apiextensions-apiserver seems like it was ALWAYS broken:
k8s.io/apiextensions/ doesn't exist, but k8s.io/apiextensions-apiserver
does.
Fixing that causes different openapi results, obviously.
Here's what others in our ecosystem are doing:
https://cs.k8s.io/?q=GINKGO_POLL_PROGRESS_(AFTER%7CINTERVAL)&i=nope&files=&excludeFiles=&repos=
the logs currently are too big partially because of this
incessant output from the progress thingy
When someone wants to debug something, they can use this
set of parameters to something lower to capture these
additional logs.
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
k8s_tag_files_matching looks for a slash after its argument, so the current value doesnt match anything
also update codegen
this is required for apiextensions-apiserver tests. After fixing apiextensions server tests to use type-aware SSA (instead of erroneously using untyped SSA) there were errors since none of the apiextensions types were actually used in the openapi given to tests.
In strict mode, stylecheck complains about Convert_* and SetDefaults_*
functions in Kubernetes because they use underscores. We want to allow that to
make the functions more readable.
This moves the hack/ directory and scripts to the examples dir, which is
a distinct module. This avoids some Go unpleasantness around module
boundaries and just makes more sense.
When running this script more than once on Debian and Ubuntu, we fail to
chown -R the CERT_DIR due to this file owned by root and the CERT_DIR
owned by the unprivileged user running the script.
Let's remove the file, that is something we can always do, before
generating the certs. This fixes the problem on Debian and Ubuntu local
setups.
Signed-off-by: Rodrigo Campos <rodrigoca@microsoft.com>
This commit is the main API piece of KEP-3257 (ClusterTrustBundles).
This commit:
* Adds the certificates.k8s.io/v1alpha1 API group
* Adds the ClusterTrustBundle type.
* Registers the new type in kube-apiserver.
* Implements the type-specfic validation specified for
ClusterTrustBundles:
- spec.pemTrustAnchors must always be non-empty.
- spec.signerName must be either empty or a valid signer name.
- Changing spec.signerName is disallowed.
* Implements the "attest" admission check to restrict actions on
ClusterTrustBundles that include a signer name.
Because it wasn't specified in the KEP, I chose to make attempts to
update the signer name be validation errors, rather than silently
ignored.
I have tested this out by launching these changes in kind and
manipulating ClusterTrustBundle objects in the resulting cluster using
kubectl.
The apiextensions-apiserver itself only depends on the following runtime
libraries when linking dynamically:
```
> ldd _output/bin/apiextensions-apiserver
linux-vdso.so.1 (0x00007ffd1b39f000)
libpthread.so.0 => /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/libpthread.so.0 (0x00007fe836022000)
libc.so.6 => /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/libc.so.6 (0x00007fe835e00000)
/nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/ld-linux-x86-64.so.2 => /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib64/ld-linux-x86-64.so.2 (0x00007fe836029000)
```
We now move the apiextensions-apiserver to become a static binary as
well to achieve maximum portability.
Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
If a CRI error occurs during the terminating phase after a pod is
force deleted (API or static) then the housekeeping loop will not
deliver updates to the pod worker which prevents the pod's state
machine from progressing. The pod will remain in the terminating
phase but no further attempts to terminate or cleanup will occur
until the kubelet is restarted.
The pod worker now maintains a store of the pods state that it is
attempting to reconcile and uses that to resync unknown pods when
SyncKnownPods() is invoked, so that failures in sync methods for
unknown pods no longer hang forever.
The pod worker's store tracks desired updates and the last update
applied on podSyncStatuses. Each goroutine now synchronizes to
acquire the next work item, context, and whether the pod can start.
This synchronization moves the pending update to the stored last
update, which will ensure third parties accessing pod worker state
don't see updates before the pod worker begins synchronizing them.
As a consequence, the update channel becomes a simple notifier
(struct{}) so that SyncKnownPods can coordinate with the pod worker
to create a synthetic pending update for unknown pods (i.e. no one
besides the pod worker has data about those pods). Otherwise the
pending update info would be hidden inside the channel.
In order to properly track pending updates, we have to be very
careful not to mix RunningPods (which are calculated from the
container runtime and are missing all spec info) and config-
sourced pods. Update the pod worker to avoid using ToAPIPod()
and instead require the pod worker to directly use
update.Options.Pod or update.Options.RunningPod for the
correct methods. Add a new SyncTerminatingRuntimePod to prevent
accidental invocations of runtime only pod data.
Finally, fix SyncKnownPods to replay the last valid update for
undesired pods which drives the pod state machine towards
termination, and alter HandlePodCleanups to:
- terminate runtime pods that aren't known to the pod worker
- launch admitted pods that aren't known to the pod worker
Any started pods receive a replay until they reach the finished
state, and then are removed from the pod worker. When a desired
pod is detected as not being in the worker, the usual cause is
that the pod was deleted and recreated with the same UID (almost
always a static pod since API UID reuse is statistically
unlikely). This simplifies the previous restartable pod support.
We are careful to filter for active pods (those not already
terminal or those which have been previously rejected by
admission). We also force a refresh of the runtime cache to
ensure we don't see an older version of the state.
Future changes will allow other components that need to view the
pod worker's actual state (not the desired state the podManager
represents) to retrieve that info from the pod worker.
Several bugs in pod lifecycle have been undetectable at runtime
because the kubelet does not clearly describe the number of pods
in use. To better report, add the following metrics:
kubelet_desired_pods: Pods the pod manager sees
kubelet_active_pods: "Admitted" pods that gate new pods
kubelet_mirror_pods: Mirror pods the kubelet is tracking
kubelet_working_pods: Breakdown of pods from the last sync in
each phase, orphaned state, and static or not
kubelet_restarted_pods_total: A counter for pods that saw a
CREATE before the previous pod with the same UID was finished
kubelet_orphaned_runtime_pods_total: A counter for pods detected
at runtime that were not known to the kubelet. Will be
populated at Kubelet startup and should never be incremented
after.
Add a metric check to our e2e tests that verifies the values are
captured correctly during a serial test, and then verify them in
detail in unit tests.
Adds 23 series to the kubelet /metrics endpoint.
Currently we only cleanup on exit. Let's trap SIGINT (ctrl-c) too, so we
always cleanup everything.
Otherwise if we ctrl-c is easy to leave something running, specially if
we ctrl-c while the cleanup function is running. And when we leave
something running and don't reused the certs ($REUSE_CERTS), that is the
default, something is left running and it fails with weird ways as we
can't auth with the new certs.
Signed-off-by: Rodrigo Campos <rodrigoca@microsoft.com>
https://github.com/kubernetes/kubernetes/pull/109728 added a
golangci-strict.yaml where gingkolinter and stylecheck (some recent additions
to golangci.yaml) were missing.
To prevent such mistakes in the future, lines that are intentionally different
get annotated with a comment about golangci-strict.yaml or golangci.yaml.
Then a suitable diff command in the new verify-golangci-lint-config.sh checks
that only such lines, comments and blank lines are different.
The long-term goal is that when "make verify" is invoked in pull job, it will
also run golangci-lint with the strict configuration and write an
$ARTIFACTS/golangci-lint-githubactions.log file with GitHub actions
annotations. How to get those published for the GitHub PR is open.
When "make verify" is invoked manually or in any other job, the stricter check
will be skipped. That works because "PR_NUMBER" is only set for pre-merge
jobs (https://github.com/kubernetes/test-infra/blob/master/prow/jobs.md#job-environment-variables).
Because strict linting is still new and might not be useful without a better
UI (= GitHub annotations), this PR does not fully enable the integration into
"make verify". Instead, the new verify-golangci-lint-pr.sh is excluded from it
and needs to be run in a separate job.
It is useful to check new code with a stricter configuration because we want it
to be of higher quality. Code reviews also become easier when reviewers don't
need to point out inefficient code manually.
What exactly should be enabled is up for debate. The current config uses the
golangci-lint defaults plus everything that is enabled explicitly by the normal
golangci.yaml, just to be on the safe side.
go list -find takes ~60% the time:
$ time go list -e ./... | grep -E -v "/(build|third_party|vendor|staging|clientset_generated|hack)/" | md5sum
b5593b3f51f3b3cd08c33bbff9627d10 -
real 0m2.687s
user 0m3.624s
sys 0m1.552s
$ time go list -find -e ./... | grep -E -v "/(build|third_party|vendor|staging|clientset_generated|hack)/" | md5sum
b5593b3f51f3b3cd08c33bbff9627d10 -
real 0m1.721s
user 0m1.675s
sys 0m1.197s
https://github.com/kubernetes/kubernetes/pull/116166#discussion_r1123924871
To run just a specific linter based on command line flags, the default
configuration needs to be disabled (because it would enable additional ones)
and then command line flags must be passed through to "golangci-lint run".
For example, to lint with just "go vet" in verbose mode, use:
verify-golangci-lint.sh -c none -- --disable-all --enable=govet -v
LC_ALL is always wanted and GREP_OPTIONS is never wanted. The `grep
--color=never` dates back to 2016, an issue with OLD grep on Macs, which
was hard to deal with when this was all Makefile magic. Now that it's a
script, we can do it simpler.
Because the script now explicitly selects the configuration file, the files no
longer have to be in the root directory. Having them in hack without the
leading dot is better because they then have the same owners as the script and
are more visible.
The downside is that manual invocations of golangci-lint without the parameter
no longer work.
All wrappers except for ExpectNoError are identical to their gomega
counterparts. The only advantage that they have is that their invocations are
shorter.
That advantage does not outweigh their disadvantages:
- cannot be used in combination with gomega.Eventually/Consistently
- not a full replacement for gomega, so we just end up using both
- don't support passing a stack offset and thus cannot be used in helper
functions
- ginkgolinter does not work for them, so sub-optimal calls like this one
are not reported:
framework.ExpectEqual(len(items), 0)
->
gomega.Expect(items).To(gomega.BeEmpty())
- developers try to make do with what's available in the framework, leading
to sub-optimal checks like this:
framework.ExpectEqual(true, strings.Contains(event.Message, expectedEventError), "Event error should indicate non-root policy caused container to not start")
->
gomega.Expect(event.Message).To(gomega.ContainSubstring(expectedEventError), "Event error should indicate non-root policy caused container to not start")
So let's remove these wrappers. As a first step they get marked as deprecated.
This enables stricter
linting (https://github.com/kubernetes/kubernetes/pull/109728), once enabled,
to report new code which uses them.
we should only use this env var for `arm`, since `arm64` is fully
supported by etcd folks, let us drop this!
(ex - https://github.com/etcd-io/etcd/releases/tag/v3.5.6)
ppc64le comment should be dropped as well
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
Several files that I maintain (primarily logcheck.conf right now, but
I'm also working on golangci-lint) are in this directory. It would be
useful to have approval rights to avoid delays.
The previous approach was based on the observation that some Prow jobs use the
--report-dir parameter instead of the E2E_REPORT_DIR env variable. Parsing the
command line was necessary to use the --json-report and --junit-report
parameters.
But that is complex and can be avoided by triggering the creation of complete
reports in the E2E test suite. The paths are hard-coded and relative to the
report directory to keep the code simple.
There was a report that k8s-triage started processing more data after
6db4b741dd was merged. It's unclear whether
that was because of the new <report-dir>/ginkgo_report.xml file. To avoid
this potential problem, the reports are now in a "ginkgo" sub-directory.
While at it, error checking gets enhanced:
- Create directories at the start of
the suite and bail out early if that fails.
- *All* e2e suites using the framework do this, not just test/e2e.
- Added missing error checking of truncated JUnit report writing.
Bump golangci-lint version. This version
adds support for go1.20.
This consequently also bumps the version of
staticcheck. Note that this was changed to
use master to be compatible with 1.20. This
commit reverts back to using a tagged release.
Signed-off-by: Madhav Jivrajani <madhav.jiv@gmail.com>
Swagger "docs" are actually Go code, which is used by other codegen
tools, so if you really want to regen EVERYTHING, this is part of it and
sequence matters.
Calling update-codegen.sh with no arguments runs all the functions in
definition order. Client-generation depends on protobuf, so protobuf
codegen needs to be near the beginning.
Also add some debug output for protobuf generation.
Also hide some old, verbose debug output.
Now update-generated-proto-bindings rules all the api.pb.go generation.
Running this shows no delta on the runtime.pb.go
This exposes an issue in how protoc is called for protos that specify
`go_package` which is fixed here.
Not all of our protos specify that option (even though it is
recommended), which will be fixed subsequently.
Each of these scripts is basically identical, and all were too brittle.
Now they should be more resilient and easier to manage. The script
still needs to be updated if we add new ones, which I do not love.
More cleanup to follow.
The `find` tool has hard to comprehend syntax and does not consider
things excluded by .gitignore. I keep tripping over this in my own
repos, where I have __stuff which gets found.
This converts update-codegen to use `git ls-files` in a seemingly
equivalent way (`-cmo --exclude-standard`). I verified it finds the
same set of files as before.
This also drops some obsolete filtering.
Also hide grep errors for not-found files, which can happen if a file is
removed but git ls-files still knows it.
Re-running update-codegen shows no diffs.
This will make subsequent changes easier.
Don't just grep for DO NOT EDIT - anchor it in something that looks like
a comment and alone on a line.
Also ignore __* dirs
Prevent it from triggering on update-generated-swagger-docs (hack, but
better than before)
The env vars are needed until go workspaces lands, then it can get
simpler.
Downsides to this:
1) If you don't call kube::golang::setup_env, it might work but will
just splat results somewhere
2) The resultant binaries are not in _output/bin but instead in the
phony GOPATH/bin (which setup_env puts in PATH)
hack/pin-dependency.sh github.com/moby/ipvs v1.1.0
- go to a fixed tag for `vishvananda/netns`
- no more references to `pkg/errors`
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
This came up when updating go-oidc. After updating go-oidc (with its
dependency tree), cloud.google.com/go was no longer used as a package
import, but still listed in the module dependency graph; as a result,
"go mod vendor" no longer pulled in cloud.google.com/go itself, but
update-vendor-licenses.sh still wanted a license file for it since it
appeared in the list of modules.
This scenario is already supposed to be handled: when a module doesn't
contain any *files* as first-level content, if the number of
subdirectories it contains *equals* the number of submodules it
contains (excluding itself), the module is skipped. This fails for
cloud.google.com/go because several submodules are included in the
module dependency graph but aren't actually used, and therefore not
vendored.
Updating the test to check that the number of subdirectories is less
than or equal to the number of expected submodules fixes this.
The correct fix would be to process the submodules first, keeping a
note of which ones really have content, then check that the top-level
module only contains subdirectories corresponding to those modules;
but it's not clear to me that this is worth the effort (especially in
a shell script).
Signed-off-by: Stephen Kitt <skitt@redhat.com>
From the warning message that ginkgo now emits:
--slow-spec-threshold is deprecated --slow-spec-threshold has been deprecated
and will be removed in a future version of Ginkgo. This feature has proved
to be more noisy than useful. You can use --poll-progress-after, instead, to
get more actionable feedback about potentially slow specs and understand
where they might be getting stuck.
We already use --poll-progress-after.
This is a fix for 104aab81a4: because
the default was not set for E2E_TEST_DEBUG_TOOL, all parameters were always
also passed to the E2E suite.
That wasn't wrong for the parameters so far, but breaks when using something
like --output-dir which is only understood by the CLI.
If the script was called with no arguments, it passed "${@:-}" to the suite,
which expands to one empty argument. That's not right, "${@}" should be used
instead because it expands to nothing when empty.
Every ginkgo callback should return immediately when a timeout occurs or the
test run manually gets aborted with CTRL-C. To do that, they must take a ctx
parameter and pass it through to all code which might block.
This is a first automated step towards that: the additional parameter got added
with
sed -i 's/\(framework.ConformanceIt\|ginkgo.It\)\(.*\)func() {$/\1\2func(ctx context.Context) {/' \
$(git grep -l -e framework.ConformanceIt -e ginkgo.It )
$GOPATH/bin/goimports -w $(git status | grep modified: | sed -e 's/.* //')
log_test.go was left unchanged.
The kubectl-convert binary itself only depends on the following runtime
libraries when linking dynamically:
```
> ldd _output/bin/kubectl-convert
linux-vdso.so.1 (0x00007ffef0786000)
libpthread.so.0 => /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/libpthread.so.0 (0x00007f5f4ac25000)
libdl.so.2 => /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/libdl.so.2 (0x00007f5f4ac20000)
libc.so.6 => /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/libc.so.6 (0x00007f5f4aa00000)
/nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/ld-linux-x86-64.so.2 => /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib64/ld-linux-x86-64.so.2 (0x00007f5f4ac2c000)
```
We now move kubectl-convert to become a static binary as well to achieve
maximum portability.
Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
The kube-aggregator itself only depends on the following runtime
libraries when linking dynamically:
```
> ldd _output/bin/kube-aggregator
linux-vdso.so.1 (0x00007fff1616f000)
libpthread.so.0 => /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/libpthread.so.0 (0x00007fad9339a000)
libc.so.6 => /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/libc.so.6 (0x00007fad93000000)
/nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/ld-linux-x86-64.so.2 => /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib64/ld-linux-x86-64.so.2 (0x00007fad933a1000)
```
We now move the kube-aggregator to become a static binary as well to
achieve maximum portability.
Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
We need to better catch problems like the one we fixed in:
7b6a3d6a33
Essentially a copy-paste error where had a branch pointing to staging/src/k8s.io/code-generator instead of staging/src/k8s.io/apimachinery
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
The call to `i18n.LoadTranslations` needs to occur on init so
that package-level variables that call `i18n.T()` to initialize
their values will be able to receive the translated string.
Added new integration tests to test help output translation.
github.com/container-orchestrated-devices/container-device-interface and
github.com/opencontainers/runtime-tools depend on some problematic packages in
code that isn't used by Kubernetes. Explicitly listing those dependencies
ensures that they do not get pulled in accidentally.
Dependencies need to be updated to use
github.com/container-orchestrated-devices/container-device-interface.
It's not decided yet whether we will implement Topology support
for DRA or not. Not having any toppology-related code
will help to avoid wrong impression that DRA is used as a hint
provider for the Topology Manager.
This adds a new resource.k8s.io API group with v1alpha1 as version. It contains
four new types: resource.ResourceClaim, resource.ResourceClass, resource.ResourceClaimTemplate, and
resource.PodScheduling.
This was originally added in the early days of go.mod in order to ensure that the root k8s.io/kubernetes
go.mod file reflected all the versions of dependencies we were using, and to ensure that go commands didn't spuriously
increment these versions.
With module pruning in go1.17+ and go defaulting to mod=readonly mode in go1.16+, this is no longer necessary.
Most parameters can be passed to both the CLI and the suite, but some
(for example, --ginkgo.slow-spec-threshold) had no effect when only
passed to the suite.
DOCKER is otherwise used to be the command name (perhaps podman), but we were conflating DOCKER_OPTS in kube::util::ensure_docker_daemon_connectivity.
Split out docker opts.
This fixes shellcheck warning that docker is assigned an array and then a string in some scripts.
Adding the ability to ignore no schedule flags in testing.
Specifically node.cloudprovider.kubernetes.io/uninitialized:NoSchedule
Fix shellcheck complaint.
The `make` rules which auto-generate some of our API stuff are
incredibly baroque, and hard to maintain. They were originally added on
the assumption that we would stop checking generated files into git.
Since then we have moved away from that goal, and the worst problems
with generated files have been resolved.
Reasons to kill this:
* It is slow on every build, as opposed to just being slow when running
the generators. It is even slow to calculate that there's nothing to
update.
* Most development work doesn't involve changing APIs.
* It only covers about half (or less) of the generated code, and making
it cover more would be even slower.
* Approximately 1 person knows how this all works.
* We have CI to make sure changes do not get merged without updating
this code.
* We have corner cases where this does the WRONG thing and tracking
those down is ugly and hard in perpetuity.
So this commit puts all the same logic that WAS in the
Makefile.generated_files into update-codegen.sh.
I do not love this script, especially WRT sub-packages, but I am trying
not to boil the ocean. I hope to follow up with some more cleanups over
time.
I have tested this manually and with the scripts and it still seems to
catch errors properly.
This includes a change to kube::util::read-array to make it not unset
variables and not over-write non-array variables.
A staging repo which just got created with only the doc.go file in it won't
have any dependencies yet, which caused the script to fail because the
dependency files didn't get created:
+++ [0926 14:33:22] go.mod: tidying
cat: /tmp/update-vendor.1VTv/group_replace.ZbIT/go.mod.require_direct.tmp: No such file or directory
!!! [0926 14:33:23] Call tree:
!!! [0926 14:33:23] 1: hack/update-vendor.sh:354 group_directives(...)
- Moves kms proto apis to the staging repo
- Updates generate and verify kms proto scripts to check staging repo
Signed-off-by: Anish Ramasekar <anish.ramasekar@gmail.com>
The new support in ginkgo for progress reports while a test runs dumps
information about where a test is stuck when it runs too long. This can provide
additional insights into what the test is waiting for.
For the Kubernetes jobs using ginkgo-e2e.sh, such dumps are now enabled after
300 seconds and then get repeated every 20 seconds. The initial delay is
intentionally the same as for warning about a slow test. The rationale is that
such test runtimes are unexpected and may need further information to diagnose
why they are slow.
With -ginkgo.source-root, Ginkgo is able to locate the Kubernetes source code
and display small source code snippets for functions that are related to the
test, determined through a heuristic that assumes that all files under the test
suite are for the tests in it.
Set intercept mode to none will help to reveal more information when the
test hangs,
- https://github.com/onsi/ginkgo/issues/970
or circumvent cases where the code grabbing the stdout/stderr pipe
is not under the framework control and may cause hangs,
- https://github.com/onsi/ginkgo/issues/851
The flag `output-interceptor-mode` is set to `none` as we were trying to
figure out of the rootcase of the test flaky, it's only intended for debugging.
- https://github.com/kubernetes/kubernetes/issues/111086
But this set also has some side effect, since it will turn off stdout/stderr
capture completely, any output to stdout/stderr will be lost.
Now that the root cause is not caused by Ginkgo bump nor how the intercept
mode was set, we'd better to follow the default value.
Signed-off-by: Dave Chen <dave.chen@arm.com>
Some scripts and tools still relied on the deprecated flags, the ones
which are about to be removed.
This is intentionally not a complete removal of all those flags in the entire
repo. This would lead to much more code churn also in places where commands
still accept the flags because they use klog directly.
Introduce networking/v1alpha1 api group.
Add `ClusterCIDR` type to networking/v1alpha1 api group, this type
will enable the NodeIPAM controller to support multiple ClusterCIDRs.
- add feature gate
- add encrypted object and run generated_files
- generate protobuf for encrypted object and add unit tests
- move parse endpoint to util and refactor
- refactor interface and remove unused interceptor
- add protobuf generate to update-generated-kms.sh
- add integration tests
- add defaulting for apiVersion in kmsConfiguration
- handle v1/v2 and default in encryption config parsing
- move metrics to own pkg and reuse for v2
- use Marshal and Unmarshal instead of serializer
- add context for all service methods
- check version and keyid for healthz
Signed-off-by: Anish Ramasekar <anish.ramasekar@gmail.com>
This change is to promote local storage capacity isolation feature to GA
At the same time, to allow rootless system disable this feature due to
unable to get root fs, this change introduced a new kubelet config
"localStorageCapacityIsolation". By default it is set to true. For
rootless systems, they can set this configuration to false to disable
the feature. Once it is set, user cannot set ephemeral-storage
request/limit because capacity and allocatable will not be set.
Change-Id: I48a52e737c6a09e9131454db6ad31247b56c000a
This applies to all jobs using hack/ginkgo-e2e.sh. This is done because
Spyglass does not render the escape sequences, making test output harder to
read.
It is done here because then we don't need to set GINKGO_NO_COLOR in all the
different Prow job configs.
Ginkgo v1 had a much longer default test timeout, in v2 this
switched to being 1 hour. This is not long enough to run many of our
suites.
Here we copy the backwards compatibility that is used by
hack/gingo-e2e.sh to unbreak serial pipelines.
Ginkgo has been migrated to V2, add this to unwanted dependencies
so that it won't be shown up as a dep again in the future.
Signed-off-by: Dave Chen <dave.chen@arm.com>
The alias for vendor/github.com/onsi/ginkgo/ginkgo ensures that code like
30e99cb2a9/experiment/kind-conformance-image-e2e.sh (L110)
continues to work. The one without "vendor/" is there just in case that it
was used because it also worked.
Long term, "ginkgo" is a nicer, version independent alias. It gets used
internally to avoid future churn and gets documented also publicly in the
Makefile help.
The caveat is that there's no guarantee that a future v3 CLI will be compatible
with current invocations. But the most common usage is through
hack/ginkgo-e2e.sh, which can deal with such differences.
Default timeout setting has been reduced from `24h` down to `1h` in
Ginkgo V2, but for some long running test this is too short.
How long to abort the test was controlled by the the linux command `timeout`
in V1. e.g. `'timeout -k 30s 150m ...`, and is configured in the file
like `sig-network-misc.yaml`.
Set the timeout manually for Ginkgo V2 to avoid the early aborting.
Signed-off-by: Dave Chen <dave.chen@arm.com>
The change is needed for `verify-e2e-test-ownership.sh`.
The `jq` is re-defined since the structure of test spec
is different with v1 and the stacktrace related validation
is not available, e.g. `package` and `func`.
Signed-off-by: Dave Chen <dave.chen@arm.com>
The test/e2e directory contains several unit tests that should run as part of
"make test":
./test/e2e/chaosmonkey/chaosmonkey_test.go
./test/e2e/storage/external/external_test.go
./test/e2e/storage/utils/utils_test.go
./test/e2e/framework/log_test.go
./test/e2e/framework/testfiles/testfiles_test.go
./test/e2e/framework/timer/timer_test.go
./test/e2e/framework/node/wait_test.go
./test/e2e/framework/pod/resource_test.go
./test/e2e/framework/config/config_test.go
./test/e2e/framework/ingress/ingress_utils_test.go
./test/e2e/framework/providers/gce/firewall_test.go
Because they were excluded by "./test/e2e/*", some of them became outdated.
./test/e2e/e2e_test.go is the only test that needs to be excluded because it is
the E2E test suite that depends on a functional cluster.
The following investigation occurred during development.
Add TimingHistogram impl that shares lock with WeightedHistogram
Benchmarking and profiling shows that two layers of locking is
noticeably more expensive than one.
After adding this new alternative, I now get the following benchmark
results.
```
(base) mspreitz@mjs12 kubernetes % go test -benchmem -run=^$ -bench ^BenchmarkTimingHistogram$ k8s.io/component-base/metrics/prometheusextension
goos: darwin
goarch: amd64
pkg: k8s.io/component-base/metrics/prometheusextension
cpu: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
BenchmarkTimingHistogram-16 22232037 52.79 ns/op 0 B/op 0 allocs/op
PASS
ok k8s.io/component-base/metrics/prometheusextension 1.404s
(base) mspreitz@mjs12 kubernetes % go test -benchmem -run=^$ -bench ^BenchmarkTimingHistogram$ k8s.io/component-base/metrics/prometheusextension
goos: darwin
goarch: amd64
pkg: k8s.io/component-base/metrics/prometheusextension
cpu: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
BenchmarkTimingHistogram-16 22190997 54.50 ns/op 0 B/op 0 allocs/op
PASS
ok k8s.io/component-base/metrics/prometheusextension 1.435s
```
and
```
(base) mspreitz@mjs12 kubernetes % go test -benchmem -run=^$ -bench ^BenchmarkTimingHistogramDirect$ k8s.io/component-base/metrics/prometheusextension
goos: darwin
goarch: amd64
pkg: k8s.io/component-base/metrics/prometheusextension
cpu: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
BenchmarkTimingHistogramDirect-16 28863244 40.99 ns/op 0 B/op 0 allocs/op
PASS
ok k8s.io/component-base/metrics/prometheusextension 1.890s
(base) mspreitz@mjs12 kubernetes %
(base) mspreitz@mjs12 kubernetes %
(base) mspreitz@mjs12 kubernetes % go test -benchmem -run=^$ -bench ^BenchmarkTimingHistogramDirect$ k8s.io/component-base/metrics/prometheusextension
goos: darwin
goarch: amd64
pkg: k8s.io/component-base/metrics/prometheusextension
cpu: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
BenchmarkTimingHistogramDirect-16 27994173 40.37 ns/op 0 B/op 0 allocs/op
PASS
ok k8s.io/component-base/metrics/prometheusextension 1.384s
```
So the new implementation is roughly 20% faster than the original.
Add overlooked exception, rename timingHistogram to timingHistogramLayered
Use the direct (one mutex) style of TimingHistogram impl
This is about a 20% gain in CPU speed on my development machine, in
benchmarks without lock contention. Following are two consecutive
trials.
(base) mspreitz@mjs12 prometheusextension % go test -benchmem -run=^$ -bench Histogram .
goos: darwin
goarch: amd64
pkg: k8s.io/component-base/metrics/prometheusextension
cpu: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
BenchmarkTimingHistogramLayered-16 21650905 51.91 ns/op 0 B/op 0 allocs/op
BenchmarkTimingHistogramDirect-16 29876860 39.33 ns/op 0 B/op 0 allocs/op
BenchmarkWeightedHistogram-16 49227044 24.13 ns/op 0 B/op 0 allocs/op
BenchmarkHistogram-16 41063907 28.82 ns/op 0 B/op 0 allocs/op
PASS
ok k8s.io/component-base/metrics/prometheusextension 5.432s
(base) mspreitz@mjs12 prometheusextension % go test -benchmem -run=^$ -bench Histogram .
goos: darwin
goarch: amd64
pkg: k8s.io/component-base/metrics/prometheusextension
cpu: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
BenchmarkTimingHistogramLayered-16 22483816 51.72 ns/op 0 B/op 0 allocs/op
BenchmarkTimingHistogramDirect-16 29697291 39.39 ns/op 0 B/op 0 allocs/op
BenchmarkWeightedHistogram-16 48919845 24.03 ns/op 0 B/op 0 allocs/op
BenchmarkHistogram-16 41153044 29.26 ns/op 0 B/op 0 allocs/op
PASS
ok k8s.io/component-base/metrics/prometheusextension 5.044s
Remove layered implementation of TimingHistogram
This commit cleans up references to the old kubernetes-node-e2e-images
project. In the process it removes the `LIST_IMAGES` mode as listing
large numbers of public cloud projects is not particularly useful, and
has been somewhat broken for a long period of time - as we defaulted
launching a VM to a different project than listing.
This commit undoes the GODEBUG=x509sha1=1 workaround.
The problem should be fixed in Go 1.18.1 now.
Signed-off-by: Madhav Jivrajani <madhav.jiv@gmail.com>
* Introduce networking/v1alpha1 api, ClusterCIDRConfig type
Introduce networking/v1alpha1 api group.
Add `ClusterCIDRConfig` type to networking/v1alpha1 api group, this type
will enable the NodeIPAM controller to support multiple ClusterCIDRs.
* Change ClusterCIDRConfig.NodeSelector type in api
* Fix review comments for API
* Update ClusterCIDRConfig API Spec
Introduce PerNodeHostBits field, remove PerNodeMaskSize
Over time the size of our junit xml has exploded to the point where
test-grid fails to process them. We still have the original/full
*.stdout files from where the junit xml files are generated from so the
junit xml files need NOT have the fill/exact output for
processing/display. So let us prune the large messages with an
indicator that we have "[... clipped...]" some of the content so folks
can see that they have to consult the full *.stdout files.
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
