kubernetes/hack
Clayton Coleman 6b9a381185
kubelet: Force deleted pods can fail to move out of terminating
If a CRI error occurs during the terminating phase after a pod is
force deleted (API or static) then the housekeeping loop will not
deliver updates to the pod worker which prevents the pod's state
machine from progressing. The pod will remain in the terminating
phase but no further attempts to terminate or cleanup will occur
until the kubelet is restarted.

The pod worker now maintains a store of the pods state that it is
attempting to reconcile and uses that to resync unknown pods when
SyncKnownPods() is invoked, so that failures in sync methods for
unknown pods no longer hang forever.

The pod worker's store tracks desired updates and the last update
applied on podSyncStatuses. Each goroutine now synchronizes to
acquire the next work item, context, and whether the pod can start.
This synchronization moves the pending update to the stored last
update, which will ensure third parties accessing pod worker state
don't see updates before the pod worker begins synchronizing them.

As a consequence, the update channel becomes a simple notifier
(struct{}) so that SyncKnownPods can coordinate with the pod worker
to create a synthetic pending update for unknown pods (i.e. no one
besides the pod worker has data about those pods). Otherwise the
pending update info would be hidden inside the channel.

In order to properly track pending updates, we have to be very
careful not to mix RunningPods (which are calculated from the
container runtime and are missing all spec info) and config-
sourced pods. Update the pod worker to avoid using ToAPIPod()
and instead require the pod worker to directly use
update.Options.Pod or update.Options.RunningPod for the
correct methods. Add a new SyncTerminatingRuntimePod to prevent
accidental invocations of runtime only pod data.

Finally, fix SyncKnownPods to replay the last valid update for
undesired pods which drives the pod state machine towards
termination, and alter HandlePodCleanups to:

- terminate runtime pods that aren't known to the pod worker
- launch admitted pods that aren't known to the pod worker

Any started pods receive a replay until they reach the finished
state, and then are removed from the pod worker. When a desired
pod is detected as not being in the worker, the usual cause is
that the pod was deleted and recreated with the same UID (almost
always a static pod since API UID reuse is statistically
unlikely). This simplifies the previous restartable pod support.
We are careful to filter for active pods (those not already
terminal or those which have been previously rejected by
admission). We also force a refresh of the runtime cache to
ensure we don't see an older version of the state.

Future changes will allow other components that need to view the
pod worker's actual state (not the desired state the podManager
represents) to retrieve that info from the pod worker.

Several bugs in pod lifecycle have been undetectable at runtime
because the kubelet does not clearly describe the number of pods
in use. To better report, add the following metrics:

  kubelet_desired_pods: Pods the pod manager sees
  kubelet_active_pods: "Admitted" pods that gate new pods
  kubelet_mirror_pods: Mirror pods the kubelet is tracking
  kubelet_working_pods: Breakdown of pods from the last sync in
    each phase, orphaned state, and static or not
  kubelet_restarted_pods_total: A counter for pods that saw a
    CREATE before the previous pod with the same UID was finished
  kubelet_orphaned_runtime_pods_total: A counter for pods detected
    at runtime that were not known to the kubelet. Will be
    populated at Kubelet startup and should never be incremented
    after.

Add a metric check to our e2e tests that verifies the values are
captured correctly during a serial test, and then verify them in
detail in unit tests.

Adds 23 series to the kubelet /metrics endpoint.
2023-03-08 22:03:51 -06:00
..
boilerplate Make boilerplate.py smarter about generated 2023-01-21 11:16:13 -08:00
conformance e2e: accept context from Ginkgo 2022-12-10 19:50:18 +01:00
e2e-internal hack/update-bazel.sh 2021-02-28 15:17:29 -08:00
gen-swagger-doc
jenkins Defer builds to test-cmd and test-integration targets 2023-02-01 15:35:14 -05:00
lib Remove unnecessary ETCD_UNSUPPORTED_ARCH for arm64 2023-02-16 21:29:13 -05:00
make-rules Merge pull request #115246 from thockin/codegen-11-swagger-from-update-codegen 2023-01-22 11:24:10 -08:00
testdata Updating pause image refernces to 3.9 2022-11-14 10:24:54 -08:00
tools hack/tools: bump golangci-lint to v1.51.2 2023-02-22 07:54:20 +01:00
verify-flags Update sig-cli OWNERS 2022-02-14 10:55:35 -07:00
.descriptions_failures Add missing comments in APF API types 2021-06-25 00:27:40 -04:00
.import-aliases Adding new api version of admissionregistration.k8s.io v1alpha1 for CEL in Admission Control 2022-11-07 20:51:51 +00:00
.spelling_failures Add gimme 2023-02-01 16:34:23 -05:00
benchmark-go.sh
build-cross.sh
build-go.sh
cherry_pick_pull.sh Update cherry_pick_pull.sh 2022-06-26 09:21:05 +05:30
dev-build-and-push.sh
dev-build-and-up.sh
dev-push-conformance.sh Build Ginkgo binary 2022-07-08 10:46:11 +08:00
e2e-node-test.sh
generate-docs.sh
get-build.sh cluster,hack: Use community infra GCS bucket for retrieving CI builds 2021-07-03 17:04:54 -04:00
ginkgo-e2e.sh e2e: revise complete report creation 2023-02-10 10:20:20 +01:00
golangci.yaml golangci-lint: update logcheck original-url 2023-02-28 17:22:33 +01:00
grab-profiles.sh Fix exit code check in hack/grab-profiles.sh 2021-05-17 14:47:05 +08:00
install-etcd.sh
install-protoc.sh Add helper script to install protoc 2023-01-26 18:00:08 -05:00
lint-dependencies.sh Guard against merging non-canonical replace directives 2022-11-17 13:00:12 -05:00
list-feature-tests.sh
local-up-cluster.sh Fix lint errors at local-up-cluster.sh 2023-01-18 12:20:33 +01:00
logcheck.conf Add ctx logging involved in KS startup (#115588) 2023-02-13 09:19:29 -08:00
module-graph.sh
OWNERS Move root approvers to subdirs 2022-10-10 13:43:03 -04:00
pin-dependency.sh Stop adding explicit replace directives for all requires 2022-10-28 15:15:18 -04:00
print-workspace-status.sh Remove invalid comments in hack/lib/version.sh 2022-07-06 19:19:29 +08:00
README.md
run-in-gopath.sh
run-prometheus-on-etcd-scrapes.sh Add serving of scrapes as Prometheus metrics 2021-11-08 15:28:05 -05:00
serve-prom-scrapes.sh Fixed typo in hack/serve-prom-scrapes.sh 2022-02-04 01:33:06 -05:00
test-go.sh
test-integration.sh
unwanted-dependencies.json upgrade etcd deps to v3.5.7 2023-03-01 12:14:52 +08:00
update-all.sh refactor(hack): use ${BASH_SOURCE[0]} to get script name 2022-04-16 23:58:13 +08:00
update-codegen.sh Merge pull request #116141 from thockin/codegen_script_wide_vars 2023-02-28 14:54:23 -08:00
update-conformance-yaml.sh port conformance yaml generation to hack 2021-02-28 00:54:54 -08:00
update-generated-api-compatibility-data.sh Add CRD compatibility objects 2023-02-27 14:12:59 -05:00
update-generated-docs.sh
update-generated-proto-bindings-dockerized.sh Check protoc version strictly 2023-01-26 18:00:06 -05:00
update-generated-protobuf-dockerized.sh Check protoc version strictly 2023-01-26 18:00:06 -05:00
update-generated-stable-metrics.sh remove the rest of the bazel test wiring for metrics stability 2021-02-05 10:48:48 -08:00
update-gofmt.sh Make update-gofmt use ls-files 2023-01-22 15:16:23 -08:00
update-import-aliases.sh
update-internal-modules.sh Rename _examples to examples 2021-01-25 10:20:46 -08:00
update-kustomize.sh kubectl version should include bundled kustomize version 2022-03-24 13:31:57 -04:00
update-mocks.sh Fix update-mocks to use better globs 2023-01-23 09:20:12 -08:00
update-netparse-cve.sh Make update-netparse-cve use ls-files 2023-01-22 15:21:37 -08:00
update-openapi-spec.sh Remove openapi files before regen 2023-01-21 17:46:22 -08:00
update-translations.sh fix translations location in update-translations.sh 2021-07-07 20:01:25 +02:00
update-vendor-licenses.sh Licensing: skip modules with fewer subdirs than mods 2023-01-03 16:48:35 +01:00
update-vendor.sh Stop adding requires for everything in the graph 2022-10-28 15:15:18 -04:00
update-yamlfmt.sh Make verify-yamlfmt.sh use git worktree 2022-12-26 17:14:58 -08:00
verify-all.sh refactor(hack): use ${BASH_SOURCE[0]} to get script name 2022-04-16 23:58:13 +08:00
verify-api-groups.sh Add discovery types 2022-10-19 17:27:25 +00:00
verify-boilerplate.sh
verify-cli-conventions.sh
verify-codegen.sh Simplify verify-codegen to use worktrees 2023-01-21 11:16:15 -08:00
verify-conformance-requirements.sh fix make verify (#115871) 2023-02-22 07:17:56 -08:00
verify-conformance-yaml.sh port conformance yaml generation to hack 2021-02-28 00:54:54 -08:00
verify-description.sh hack/verify-description.sh: correctly look for versioned API types 2021-03-02 21:50:40 +05:30
verify-e2e-test-ownership.sh declare and assign seperately in hack/verify-e2e-test-ownership.sh 2022-10-19 23:46:18 -07:00
verify-external-dependencies-version.sh use sed to drop escape codes from zeitgeist output 2021-03-04 14:59:52 -08:00
verify-fieldname-docs.sh Fix the name violation in apiextensions.k8s.io/v1,CustomResourceConversion, remove the failures file 2023-01-19 22:08:29 +08:00
verify-flags-underscore.py use python3 in hack 2023-01-12 11:20:41 +08:00
verify-generated-docs.sh
verify-generated-stable-metrics.sh remove the rest of the bazel test wiring for metrics stability 2021-02-05 10:48:48 -08:00
verify-gofmt.sh cleanup verify-gofmt.sh 2021-03-07 12:51:33 -08:00
verify-golangci-lint.sh hack: move golangci-lint config files 2023-02-28 17:22:31 +01:00
verify-govet-levee.sh KEP-1933: add static analysis target to hack/ 2020-11-03 20:03:26 +00:00
verify-govet.sh
verify-import-aliases.sh
verify-import-boss.sh drop vendor from go install paths 2022-05-05 08:47:33 -04:00
verify-imports.sh
verify-internal-modules.sh Rename *-hack-tools.sh -> *-internal-modules.sh 2021-01-25 10:15:59 -08:00
verify-licenses.sh Merge pull request #114858 from acpana/acpana/fix-verify-licenses 2023-01-05 23:15:58 -08:00
verify-mocks.sh Make verify-mocks.sh use git worktree 2022-12-26 17:14:14 -08:00
verify-netparse-cve.sh update and verify netparse 2021-08-20 10:42:09 +02:00
verify-no-vendor-cycles.sh Update go.mod files to go1.18, update license vendor script 2022-05-04 10:22:27 -04:00
verify-non-mutating-validation.sh add verify script to catch most validation mutations 2021-10-08 14:37:55 -04:00
verify-openapi-spec.sh Generate openapi v3 static files 2021-11-16 17:38:54 -08:00
verify-pkg-names.sh
verify-prerelease-lifecycle-tags.sh Fix error path in file /verify-prerelease-lifecycle-tags.sh. 2021-03-06 15:32:07 +08:00
verify-prometheus-imports.sh kubelet: Force deleted pods can fail to move out of terminating 2023-03-08 22:03:51 -06:00
verify-publishing-bot.py use python3 in hack 2023-01-12 11:20:41 +08:00
verify-readonly-packages.sh remove clearly unnecessary lingering BUILD file references 2022-10-04 16:47:25 -07:00
verify-shellcheck.sh Update shellcheck version (0.7.2 -> 0.8.0) and fix findings 2022-11-08 11:53:57 +01:00
verify-spelling.sh
verify-staging-meta-files.sh
verify-test-code.sh e2e framework: deprecate gomega wrappers 2023-02-23 09:51:42 +01:00
verify-test-featuregates.sh
verify-test-images.sh
verify-typecheck-providerless.sh providerless tag for client-go auth plugins 2021-03-28 20:07:59 -04:00
verify-typecheck.sh Remove generated file rules in make 2022-10-04 08:50:30 -07:00
verify-vendor-licenses.sh Remove docker remote/docker-machine from build scripts 2020-12-30 12:37:51 -05:00
verify-vendor.sh Improve vendor verification works for each staging repo 2023-01-10 09:30:47 -05:00
verify-yamlfmt.sh Make verify-yamlfmt.sh use git worktree 2022-12-26 17:14:58 -08:00

Kubernetes hack GuideLines

This document describes how you can use the scripts from hack directory and gives a brief introduction and explanation of these scripts.

Overview

The hack directory contains many scripts that ensure continuous development of kubernetes, enhance the robustness of the code, improve development efficiency, etc. The explanations and descriptions of these scripts are helpful for contributors. For details, refer to the following guidelines.

Key scripts

  • verify-all.sh: This script is a vestigial redirection, Please do not add "real" logic. It is equivalent to make verify.
  • update-all.sh: This script is a vestigial redirection, Please do not add "real" logic. The true target of this makerule is hack/make-rules/update.sh.It is equivalent to make update.

Attention

Note that all scripts must be run from the Kubernetes root directory. We should run hack/verify-all.sh before submitting a PR and if anything fails run hack/update-all.sh.