github/kubernetes
4
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
b4a40cd43e
kubernetes/hack
History
Mike Spreitzer b4a40cd43e Draft weighted and timing histograms 
The following investigation occurred during development.

Add TimingHistogram impl that shares lock with WeightedHistogram

Benchmarking and profiling shows that two layers of locking is
noticeably more expensive than one.

After adding this new alternative, I now get the following benchmark
results.

```
(base) mspreitz@mjs12 kubernetes % go test -benchmem -run=^$ -bench ^BenchmarkTimingHistogram$ k8s.io/component-base/metrics/prometheusextension
goos: darwin
goarch: amd64
pkg: k8s.io/component-base/metrics/prometheusextension
cpu: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
BenchmarkTimingHistogram-16    	22232037	        52.79 ns/op	       0 B/op	       0 allocs/op
PASS
ok  	k8s.io/component-base/metrics/prometheusextension	1.404s

(base) mspreitz@mjs12 kubernetes % go test -benchmem -run=^$ -bench ^BenchmarkTimingHistogram$ k8s.io/component-base/metrics/prometheusextension
goos: darwin
goarch: amd64
pkg: k8s.io/component-base/metrics/prometheusextension
cpu: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
BenchmarkTimingHistogram-16    	22190997	        54.50 ns/op	       0 B/op	       0 allocs/op
PASS
ok  	k8s.io/component-base/metrics/prometheusextension	1.435s
```

and

```
(base) mspreitz@mjs12 kubernetes % go test -benchmem -run=^$ -bench ^BenchmarkTimingHistogramDirect$ k8s.io/component-base/metrics/prometheusextension
goos: darwin
goarch: amd64
pkg: k8s.io/component-base/metrics/prometheusextension
cpu: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
BenchmarkTimingHistogramDirect-16    	28863244	        40.99 ns/op	       0 B/op	       0 allocs/op
PASS
ok  	k8s.io/component-base/metrics/prometheusextension	1.890s
(base) mspreitz@mjs12 kubernetes %
(base) mspreitz@mjs12 kubernetes %
(base) mspreitz@mjs12 kubernetes % go test -benchmem -run=^$ -bench ^BenchmarkTimingHistogramDirect$ k8s.io/component-base/metrics/prometheusextension
goos: darwin
goarch: amd64
pkg: k8s.io/component-base/metrics/prometheusextension
cpu: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
BenchmarkTimingHistogramDirect-16    	27994173	        40.37 ns/op	       0 B/op	       0 allocs/op
PASS
ok  	k8s.io/component-base/metrics/prometheusextension	1.384s
```

So the new implementation is roughly 20% faster than the original.

Add overlooked exception, rename timingHistogram to timingHistogramLayered

Use the direct (one mutex) style of TimingHistogram impl

This is about a 20% gain in CPU speed on my development machine, in
benchmarks without lock contention.  Following are two consecutive
trials.

(base) mspreitz@mjs12 prometheusextension % go test  -benchmem -run=^$ -bench Histogram .
goos: darwin
goarch: amd64
pkg: k8s.io/component-base/metrics/prometheusextension
cpu: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
BenchmarkTimingHistogramLayered-16    	21650905	        51.91 ns/op	       0 B/op	       0 allocs/op
BenchmarkTimingHistogramDirect-16     	29876860	        39.33 ns/op	       0 B/op	       0 allocs/op
BenchmarkWeightedHistogram-16         	49227044	        24.13 ns/op	       0 B/op	       0 allocs/op
BenchmarkHistogram-16                 	41063907	        28.82 ns/op	       0 B/op	       0 allocs/op
PASS
ok  	k8s.io/component-base/metrics/prometheusextension	5.432s

(base) mspreitz@mjs12 prometheusextension % go test  -benchmem -run=^$ -bench Histogram .
goos: darwin
goarch: amd64
pkg: k8s.io/component-base/metrics/prometheusextension
cpu: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
BenchmarkTimingHistogramLayered-16    	22483816	        51.72 ns/op	       0 B/op	       0 allocs/op
BenchmarkTimingHistogramDirect-16     	29697291	        39.39 ns/op	       0 B/op	       0 allocs/op
BenchmarkWeightedHistogram-16         	48919845	        24.03 ns/op	       0 B/op	       0 allocs/op
BenchmarkHistogram-16                 	41153044	        29.26 ns/op	       0 B/op	       0 allocs/op
PASS
ok  	k8s.io/component-base/metrics/prometheusextension	5.044s

Remove layered implementation of TimingHistogram
2022-04-28 17:36:06 -04:00
..
boilerplate hack/boilerplate: Tolerate new build tag format (//go:build) 2021-08-24 15:47:49 -04:00
conformance replace ioutil with io and os for hack 2021-10-30 14:44:41 +08:00
e2e-internal hack/update-bazel.sh 2021-02-28 15:17:29 -08:00
gen-swagger-doc Bash script syntax improvements 2019-01-19 13:58:58 +01:00
jenkins hack/jenkins: set GODEBUG=x509sha1=1 2022-03-23 08:02:53 -04:00
lib Introduce APIs to support multiple ClusterCIDRs (#108290) 2022-03-30 19:39:00 -07:00
make-rules set parallelism in integration tests with GOMAXPROCS 2022-04-01 12:55:41 -04:00
testdata Switch to pause 3.7 2022-03-29 15:36:38 -04:00
tools hack: integrate logcheck into golangci-lint 2022-03-24 11:21:03 +01:00
verify-flags Update sig-cli OWNERS 2022-02-14 10:55:35 -07:00
.descriptions_failures Add missing comments in APF API types 2021-06-25 00:27:40 -04:00
.import-aliases move pkg/util/node to component-helpers/node/util (#105347) 2021-11-12 07:52:27 -08:00
.spelling_failures review updates 2019-02-22 10:00:10 -08:00
benchmark-go.sh add comment in several hack/ sh scripts. 2020-01-27 11:13:09 +09:00
build-cross.sh add comment in several hack/ sh scripts. 2020-01-27 11:13:09 +09:00
build-go.sh add comment in several hack/ sh scripts. 2020-01-27 11:13:09 +09:00
cherry_pick_pull.sh Add release note block to cherry-pick script 2021-07-28 14:46:05 -07:00
dev-build-and-push.sh fix shellcheck failure 2019-01-22 08:39:55 +08:00
dev-build-and-up.sh fix shellcheck failure 2019-01-22 08:39:55 +08:00
dev-push-conformance.sh Update references to test/conformance/image 2021-07-23 11:26:54 -04:00
e2e-node-test.sh add comment in several hack/ sh scripts. 2020-01-27 11:13:09 +09:00
generate-docs.sh fix some shellcheck failures in hack 2019-02-25 11:38:56 +08:00
get-build.sh cluster,hack: Use community infra GCS bucket for retrieving CI builds 2021-07-03 17:04:54 -04:00
ginkgo-e2e.sh Switch to use the DBG flag to build debug binaries 2022-03-09 14:57:59 +08:00
grab-profiles.sh Fix exit code check in hack/grab-profiles.sh 2021-05-17 14:47:05 +08:00
install-etcd.sh Add comments in several hack/*.sh 2020-01-28 09:27:40 +09:00
lint-dependencies.sh add update-unwanted-dependencies.sh to track unwanted dependencies 2021-09-17 13:21:38 +08:00
list-feature-tests.sh Add comments in several hack/*.sh 2020-01-28 09:27:40 +09:00
local-up-cluster.sh switch owner for CERT_DIR 2022-03-29 20:04:33 -04:00
logcheck.conf hack: integrate logcheck into golangci-lint 2022-03-24 11:21:03 +01:00
module-graph.sh Make sure the _output folder exists before using it from module-graph.sh 2020-07-07 01:19:59 -07:00
OWNERS Add enj to hack OWNERS 2022-03-02 10:04:11 -05:00
pin-dependency.sh pin-dependency.sh: enhance forking of a dependency 2021-09-14 11:30:17 +02:00
print-workspace-status.sh update print-workspace-status.sh comment 2021-03-07 12:47:37 -08:00
README.md add README.md in hack/ 2020-01-16 10:12:18 +08:00
run-in-gopath.sh Add comments in several hack/*.sh 2020-01-28 09:27:40 +09:00
run-prometheus-on-etcd-scrapes.sh Add serving of scrapes as Prometheus metrics 2021-11-08 15:28:05 -05:00
serve-prom-scrapes.sh Fixed typo in hack/serve-prom-scrapes.sh 2022-02-04 01:33:06 -05:00
test-go.sh Add comments in several hack/*.sh 2020-01-28 09:27:40 +09:00
test-integration.sh Add comments in several hack/*.sh 2020-01-28 09:27:40 +09:00
unwanted-dependencies.json Update cobra to v1.4.0 2022-03-10 16:19:43 -05:00
update-all.sh *.sh: cleanup all white noise 2020-05-29 09:56:00 +08:00
update-codegen.sh Add extract apply function generation 2021-03-10 06:13:51 -08:00
update-conformance-yaml.sh port conformance yaml generation to hack 2021-02-28 00:54:54 -08:00
update-generated-api-compatibility-data.sh Add check for unused API compatibility fixture files 2022-02-10 17:40:25 -05:00
update-generated-device-plugin-dockerized.sh add comment in hack/update-*.sh 2020-01-16 10:05:49 +08:00
update-generated-device-plugin.sh Fix shellcheck lint errors in ./hack/... 2019-02-20 01:31:58 -05:00
update-generated-docs.sh Fix shellcheck lint errors in ./hack/... 2019-02-20 01:31:58 -05:00
update-generated-kms-dockerized.sh fixing scripts from https://github.com/google/protobuf/releases to https://github.com/protocolbuffers/protobuf/releases 2021-07-29 02:24:39 +05:30
update-generated-kms.sh Fix shellcheck lint errors in ./hack/... 2019-02-20 01:31:58 -05:00
update-generated-kubelet-plugin-registration-dockerized.sh add comment in hack/update-*.sh 2020-01-16 10:05:49 +08:00
update-generated-kubelet-plugin-registration.sh Fix shellcheck lint errors in ./hack/... 2019-02-20 01:31:58 -05:00
update-generated-pod-resources-dockerized.sh Add podresources v1 API 2020-10-26 12:10:18 -07:00
update-generated-pod-resources.sh Add podresources v1 API 2020-10-26 12:10:18 -07:00
update-generated-protobuf-dockerized.sh fixing scripts from https://github.com/google/protobuf/releases to https://github.com/protocolbuffers/protobuf/releases 2021-07-29 02:24:39 +05:30
update-generated-protobuf.sh Add usage in some hack/update scripts 2020-05-26 21:49:38 -07:00
update-generated-runtime-dockerized.sh Regenerate CRI protos 2022-02-04 16:13:29 -08:00
update-generated-runtime.sh Add usage in some hack/update scripts 2020-05-26 21:49:38 -07:00
update-generated-stable-metrics.sh remove the rest of the bazel test wiring for metrics stability 2021-02-05 10:48:48 -08:00
update-generated-swagger-docs.sh Speed up update-swagger-docs 2020-08-02 10:56:46 -04:00
update-gofmt.sh fix bug of hack/update-gofmt.sh 2019-04-13 03:29:51 +08:00
update-import-aliases.sh *.sh: cleanup all white noise 2020-05-29 09:56:00 +08:00
update-internal-modules.sh Rename _examples to examples 2021-01-25 10:20:46 -08:00
update-kustomize.sh kubectl version should include bundled kustomize version 2022-03-24 13:31:57 -04:00
update-mocks.sh Added update and verify scripts for automated mock generation 2021-09-03 17:40:11 +00:00
update-netparse-cve.sh update and verify netparse 2021-08-20 10:42:09 +02:00
update-openapi-spec.sh omit enums from static openapi snapshots used to generate clients 2022-03-31 12:31:47 -04:00
update-translations.sh fix translations location in update-translations.sh 2021-07-07 20:01:25 +02:00
update-vendor-licenses.sh update license gathering script for forked code 2021-07-08 07:48:19 -04:00
update-vendor.sh Update golang.org/x/time/rate (#104014) 2021-07-29 16:13:38 -07:00
update-yamlfmt.sh Cleanup owners files to make parsing automatable 2021-12-09 21:31:03 -05:00
verify-all.sh Add comments in several hack/verify-*.sh 2020-01-16 13:54:58 +09:00
verify-api-groups.sh fix a few missed references 2020-09-02 11:45:10 -07:00
verify-boilerplate.sh Add comments in several hack/verify-*.sh 2020-01-16 13:54:58 +09:00
verify-cli-conventions.sh Add comments in several hack/verify-*.sh 2020-01-16 13:54:58 +09:00
verify-codegen.sh Drop k8s.io/node-api packages 2020-02-07 10:07:14 -05:00
verify-conformance-requirements.sh Add comments in several hack/verify-*.sh 2020-01-16 13:54:58 +09:00
verify-conformance-yaml.sh port conformance yaml generation to hack 2021-02-28 00:54:54 -08:00
verify-description.sh hack/verify-description.sh: correctly look for versioned API types 2021-03-02 21:50:40 +05:30
verify-e2e-test-ownership.sh hack/verify: add verify-e2e-test-ownership.sh 2021-03-17 16:47:17 -04:00
verify-external-dependencies-version.sh use sed to drop escape codes from zeitgeist output 2021-03-04 14:59:52 -08:00
verify-flags-underscore.py Merge pull request #92829 from marload/patch-0706-04 2020-10-19 19:12:07 -07:00
verify-generated-device-plugin.sh Add comments in several hack/verify-generated-*.sh 2020-01-17 14:20:48 +09:00
verify-generated-docs.sh Add comments in several hack/verify-generated-*.sh 2020-01-17 14:20:48 +09:00
verify-generated-files-remake.sh Always pass HEAD to worktree 2021-10-27 15:43:48 -07:00
verify-generated-files.sh Add comments in several hack/verify-generated-*.sh 2020-01-17 14:20:48 +09:00
verify-generated-kms.sh Add comments in several hack/verify-generated-*.sh 2020-01-17 14:20:48 +09:00
verify-generated-kubelet-plugin-registration.sh Add comments in several hack/verify-generated-*.sh 2020-01-17 14:20:48 +09:00
verify-generated-pod-resources.sh Add podresources v1 API 2020-10-26 12:10:18 -07:00
verify-generated-protobuf.sh add staging directories to import-boss verify script 2020-01-28 10:51:46 +01:00
verify-generated-runtime.sh Add CRI v1 proto 2020-11-10 14:17:17 -08:00
verify-generated-stable-metrics.sh remove the rest of the bazel test wiring for metrics stability 2021-02-05 10:48:48 -08:00
verify-generated-swagger-docs.sh verify-generated-swagger-docs: remove unnecessary build 2020-09-15 19:30:58 -07:00
verify-gofmt.sh cleanup verify-gofmt.sh 2021-03-07 12:51:33 -08:00
verify-golangci-lint.sh hack: integrate logcheck into golangci-lint 2022-03-24 11:21:03 +01:00
verify-govet-levee.sh KEP-1933: add static analysis target to hack/ 2020-11-03 20:03:26 +00:00
verify-govet.sh Add comments in several hack/verify-*.sh(g-r) 2020-01-20 17:10:55 +09:00
verify-import-aliases.sh Add comments in several hack/verify-*.sh(g-r) 2020-01-20 17:10:55 +09:00
verify-import-boss.sh Build code-generator tools directly 2020-08-02 10:56:46 -04:00
verify-imports.sh Add comments in several hack/verify-*.sh(g-r) 2020-01-20 17:10:55 +09:00
verify-internal-modules.sh Rename *-hack-tools.sh -> *-internal-modules.sh 2021-01-25 10:15:59 -08:00
verify-linkcheck.sh Add comments in several hack/verify-*.sh(g-r) 2020-01-20 17:10:55 +09:00
verify-mocks.sh Added update and verify scripts for automated mock generation 2021-09-03 17:40:11 +00:00
verify-netparse-cve.sh update and verify netparse 2021-08-20 10:42:09 +02:00
verify-no-vendor-cycles.sh Update verify-no-vendor-cycles to use dependencycheck 2020-08-01 08:11:05 -05:00
verify-non-mutating-validation.sh add verify script to catch most validation mutations 2021-10-08 14:37:55 -04:00
verify-openapi-spec.sh Generate openapi v3 static files 2021-11-16 17:38:54 -08:00
verify-pkg-names.sh Add comments in several hack/verify-*.sh(g-r) 2020-01-20 17:10:55 +09:00
verify-prerelease-lifecycle-tags.sh Fix error path in file /verify-prerelease-lifecycle-tags.sh. 2021-03-06 15:32:07 +08:00
verify-prometheus-imports.sh Draft weighted and timing histograms 2022-04-28 17:36:06 -04:00
verify-publishing-bot.py Change yaml.load() to safe_load() 2021-01-19 15:58:12 +10:00
verify-readonly-packages.sh Add comments in several hack/verify-*.sh(g-r) 2020-01-20 17:10:55 +09:00
verify-shellcheck.sh Fix shellcheck output streams 2021-10-14 15:53:47 +03:00
verify-spelling.sh move linting dependencies to hack/tools/tools.go 2020-04-20 13:45:38 -04:00
verify-staging-meta-files.sh Add comments in several hack/verify-*.sh(s-v) 2020-01-21 19:10:46 +09:00
verify-test-code.sh Use ExpectNoError(err) 2020-09-03 23:29:07 +00:00
verify-test-featuregates.sh Add comments in several hack/verify-*.sh(s-v) 2020-01-21 19:10:46 +09:00
verify-test-images.sh Add comments in several hack/verify-*.sh(s-v) 2020-01-21 19:10:46 +09:00
verify-typecheck-providerless.sh providerless tag for client-go auth plugins 2021-03-28 20:07:59 -04:00
verify-typecheck.sh Add env to verify-typecheck for serial execution 2022-03-09 18:21:27 -07:00
verify-vendor-licenses.sh Remove docker remote/docker-machine from build scripts 2020-12-30 12:37:51 -05:00
verify-vendor.sh Remove docker remote/docker-machine from build scripts 2020-12-30 12:37:51 -05:00
verify-yamlfmt.sh Cleanup owners files to make parsing automatable 2021-12-09 21:31:03 -05:00

README.md

Kubernetes hack GuideLines

This document describes how you can use the scripts from hack directory and gives a brief introduction and explanation of these scripts.

Overview

The hack directory contains many scripts that ensure continuous development of kubernetes, enhance the robustness of the code, improve development efficiency, etc. The explanations and descriptions of these scripts are helpful for contributors. For details, refer to the following guidelines.

Key scripts

  • verify-all.sh: This script is a vestigial redirection, Please do not add "real" logic. It is equivalent to make verify.
  • update-all.sh: This script is a vestigial redirection, Please do not add "real" logic. The true target of this makerule is hack/make-rules/update.sh.It is equivalent to make update.

Attention

Note that all scripts must be run from the Kubernetes root directory. We should run hack/verify-all.sh before submitting a PR and if anything fails run hack/update-all.sh.