Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Secure Kubelet's componentconfig defaults while maintaining CLI compatibility
This updates the Kubelet's componentconfig defaults, while applying the legacy defaults to values from options.NewKubeletConfiguration(). This keeps defaults the same for the command line and improves the security of defaults when you load config from a file.
See: https://github.com/kubernetes/kubernetes/issues/53618
See: https://github.com/kubernetes/kubernetes/pull/53833#discussion_r166669931
Also moves EnableServer to KubeletFlags, per @tallclair's comments on #53833.
We should find way of generating documentation for config file defaults, so that people can easily look up what's different from flags.
```release-note
Action required: Default values differ between the Kubelet's componentconfig (config file) API and the Kubelet's command line. Be sure to review the default values when migrating to using a config file.
```
Automatic merge from submit-queue (batch tested with PRs 59054, 59515, 59577). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add 'none' option to EnforceNodeAllocatable
This lets us use omitempty on `EnforceNodeAllocatable`. We shouldn't treat
`nil` as different from `[]T{}`, because this can play havoc with
serializers (a-la #43203).
See: https://github.com/kubernetes/kubernetes/pull/53833#discussion_r166672137
```release-note
'none' can now be specified in KubeletConfiguration.EnforceNodeAllocatable (--enforce-node-allocatable) to explicitly disable enforcement.
```
Automatic merge from submit-queue (batch tested with PRs 57824, 58806, 59410, 59280). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
2nd try at using a vanity GCR name
The 2nd commit here is the changes relative to the reverted PR. Please focus review attention on that.
This is the 2nd attempt. The previous try (#57573) was reverted while we
figured out the regional mirrors (oops).
New plan: k8s.gcr.io is a read-only facade that auto-detects your source
region (us, eu, or asia for now) and pulls from the closest. To publish
an image, push k8s-staging.gcr.io and it will be synced to the regionals
automatically (similar to today). For now the staging is an alias to
gcr.io/google_containers (the legacy URL).
When we move off of google-owned projects (working on it), then we just
do a one-time sync, and change the google-internal config, and nobody
outside should notice.
We can, in parallel, change the auto-sync into a manual sync - send a PR
to "promote" something from staging, and a bot activates it. Nice and
visible, easy to keep track of.
xref https://github.com/kubernetes/release/issues/281
TL;DR:
* The new `staging-k8s.gcr.io` is where we push images. It is literally an alias to `gcr.io/google_containers` (the existing repo) and is hosted in the US.
* The contents of `staging-k8s.gcr.io` are automatically synced to `{asia,eu,us)-k8s.gcr.io`.
* The new `k8s.gcr.io` will be a read-only alias to whichever regional repo is closest to you.
* In the future, images will be promoted from `staging` to regional "prod" more explicitly and auditably.
```release-note
Use "k8s.gcr.io" for pulling container images rather than "gcr.io/google_containers". Images are already synced, so this should not impact anyone materially.
Documentation and tools should all convert to the new name. Users should take note of this in case they see this new name in the system.
```
This is the 2nd attempt. The previous was reverted while we figured out
the regional mirrors (oops).
New plan: k8s.gcr.io is a read-only facade that auto-detects your source
region (us, eu, or asia for now) and pulls from the closest. To publish
an image, push k8s-staging.gcr.io and it will be synced to the regionals
automatically (similar to today). For now the staging is an alias to
gcr.io/google_containers (the legacy URL).
When we move off of google-owned projects (working on it), then we just
do a one-time sync, and change the google-internal config, and nobody
outside should notice.
We can, in parallel, change the auto-sync into a manual sync - send a PR
to "promote" something from staging, and a bot activates it. Nice and
visible, easy to keep track of.
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add call to addCredentialProviderFlags
**What this PR does / why we need it**:
Credential flags such as 'azure-container-registry-config' are still in use, call addCredentialProviderFlags to hook it up.
See:
https://github.com/kubernetes/kubernetes/pull/56995#issuecomment-361483382
**Which issue(s) this PR fixes**
Follow up of #56995
**Special notes for your reviewer**:
/assign @mtaufen @liggitt
**Release note**:
```release-note
NONE
```
They should both follow the convention of not being a pointer on the
internal type. This required adding a conversion function between
`int64` and `*int64`.
A side effect is this removes a warning in the generated code for the
apps API group.
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Remove deprecated --require-kubeconfig flag, remove default --kubeconfig value
**What this PR does / why we need it**:
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#41161
**Special notes for your reviewer**:
**Release note**:
```release-note
Remove deprecated --require-kubeconfig flag, remove default --kubeconfig value
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Removal of KubeletConfigFile feature gate: Step 1
This feature gate was redundant with the `--config` flag, which already
enables/disables loading Kubelet config from a file.
Since the gate guarded an alpha feature, removing it is not a violation
of our API guidelines.
Some stuff in `kubernetes/test-infra` currently sets the gate,
so removing will be a 3 step process:
1. This PR, which makes the gate a no-op.
2. Stop setting the gate in `kubernetes/test-infra`.
3. Completely remove the gate (this PR will get the release note).
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 57973, 57990). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Set pids limit at pod level
**What this PR does / why we need it**:
Add a new Alpha Feature to set a maximum number of pids per Pod.
This is to allow the use case where cluster administrators wish
to limit the pids consumed per pod (example when running a CI system).
By default, we do not set any maximum limit, If an administrator wants
to enable this, they should enable `SupportPodPidsLimit=true` in the
`--feature-gates=` parameter to kubelet and specify the limit using the
`--pod-max-pids` parameter.
The limit set is the total count of all processes running in all
containers in the pod.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#43783
**Special notes for your reviewer**:
**Release note**:
```release-note
New alpha feature to limit the number of processes running in a pod. Cluster administrators will be able to place limits by using the new kubelet command line parameter --pod-max-pids. Note that since this is a alpha feature they will need to enable the "SupportPodPidsLimit" feature.
```
This feature gate was redundant with the `--config` flag, which already
enables/disables loading Kubelet config from a file.
Since the gate guarded an alpha feature, removing it is not a violation
of our API guidelines.
Some stuff in `kubernetes/test-infra` currently sets the gate,
so removing will be a 3 step process:
1. This PR, which makes the gate a no-op.
2. Stop setting the gate in `kubernetes/test-infra`.
3. Completely remove the gate.
Add a new Alpha Feature to set a maximum number of pids per Pod.
This is to allow the use case where cluster administrators wish
to limit the pids consumed per pod (example when running a CI system).
By default, we do not set any maximum limit, If an administrator wants
to enable this, they should enable `SupportPodPidsLimit=true` in the
`--feature-gates=` parameter to kubelet and specify the limit using the
`--pod-max-pids` parameter.
The limit set is the total count of all processes running in all
containers in the pod.
Automatic merge from submit-queue (batch tested with PRs 57733, 57613, 57953). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Explicit kubelet flags
The Kubelet was using the global flag set.
Libraries also often use the global flag set.
There are flags in the Kubelet's usage statement for which the Kubelet does not control registration.
This is bad, we must have full control of our command-line API.
This PR puts a stop to it.
I worked through the Kubelet's usage printout and tracked down the flags.
In the below list, flags with checkboxes are one of:
- explicitly **registered** in this PR
- thought about and outright **rejected**
- thought about and **registered-deprecated** for legacy reasons
For the others we need to figure out whether they should be **registered** or **rejected** or **registered-deprecated**.
### cadvisor:
grep commands, for reference:
```
# flag definitions in cadvisor
git grep -E "\"application_metrics_count_limit\"|\"boot_id_file\"|\"container_hints\"|\"containerd\"|\"docker\"|\"docker_env_metadata_whitelist\"|\"docker_only\"|\"docker_root\"|\"docker-tls\"|\"docker-tls-ca\"|\"docker-tls-cert\"|\"docker-tls-key\"|\"enable_load_reader\"|\"event_storage_age_limit\"|\"event_storage_event_limit\"|\"global_housekeeping_interval\"|\"housekeeping_interval\"|\"log_cadvisor_usage\"|\"machine_id_file\"|\"storage_driver_buffer_duration\"|\"storage_driver_db\"|\"storage_driver_host\"|\"storage_driver_password\"|\"storage_driver_secure\"|\"storage_driver_table\"|\"storage_driver_user\"" -- vendor/github.com/google/cadvisor
# flag invocations
git grep -E "\--application_metrics_count_limit|--boot_id_file|--container_hints|--containerd|--docker|--docker_env_metadata_whitelist|--docker_only|--docker_root|--docker_tls|--docker_tls_ca|--docker_tls_cert|--docker_tls_key|--enable_load_reader|--event_storage_age_limit|--event_storage_event_limit|--global_housekeeping_interval|--housekeeping_interval|--log_cadvisor_usage|--machine_id_file|--storage_driver_buffer_duration|--storage_driver_db|--storage_driver_host|--storage_driver_password|--storage_driver_secure|--storage_driver_table|--storage_driver_user" -- ':(exclude)pkg/generated/bindata.go' ':(exclude)Godeps' ':(exclude)CHANGELOG*' ':(exclude)vendor'
# normalized flag invocations
git grep -E "\--application-metrics-count-limit|--boot-id-file|--container-hints|--containerd|--docker|--docker-env-metadata-whitelist|--docker-only|--docker-root|--docker-tls|--docker-tls-ca|--docker-tls-cert|--docker-tls-key|--enable-load-reader|--event-storage-age-limit|--event-storage-event-limit|--global-housekeeping-interval|--housekeeping-interval|--log-cadvisor-usage|--machine-id-file|--storage-driver-buffer-duration|--storage-driver-db|--storage-driver-host|--storage-driver-password|--storage-driver-secure|--storage-driver-table|--storage-driver-user" -- ':(exclude)pkg/generated/bindata.go' ':(exclude)Godeps' ':(exclude)CHANGELOG*' ':(exclude)vendor'
# flag names
git grep -E "application_metrics_count_limit|boot_id_file|container_hints|containerd|docker|docker_env_metadata_whitelist|docker_only|docker_root|docker_tls|docker_tls_ca|docker_tls_cert|docker_tls_key|enable_load_reader|event_storage_age_limit|event_storage_event_limit|global_housekeeping_interval|housekeeping_interval|log_cadvisor_usage|machine_id_file|storage_driver_buffer_duration|storage_driver_db|storage_driver_host|storage_driver_password|storage_driver_secure|storage_driver_table|storage_driver_user" ':(exclude)pkg/generated/bindata.go' ':(exclude)Godeps' ':(exclude)CHANGELOG*' ':(exclude)vendor'
# normalized flag names
git grep -E "application-metrics-count-limit|boot-id-file|container-hints|containerd|docker|docker-env-metadata-whitelist|docker-only|docker-root|docker-tls|docker-tls-ca|docker-tls-cert|docker-tls-key|enable-load-reader|event-storage-age-limit|event-storage-event-limit|global-housekeeping-interval|housekeeping-interval|log-cadvisor-usage|machine-id-file|storage-driver-buffer-duration|storage-driver-db|storage-driver-host|storage-driver-password|storage-driver-secure|storage-driver-table|storage-driver-user" ':(exclude)pkg/generated/bindata.go' ':(exclude)Godeps' ':(exclude)CHANGELOG*' ':(exclude)vendor'
# normalized and underscore names combined
git grep -E "application_metrics_count_limit|boot_id_file|container_hints|containerd|docker|docker_env_metadata_whitelist|docker_only|docker_root|docker_tls|docker_tls_ca|docker_tls_cert|docker_tls_key|enable_load_reader|event_storage_age_limit|event_storage_event_limit|global_housekeeping_interval|housekeeping_interval|log_cadvisor_usage|machine_id_file|storage_driver_buffer_duration|storage_driver_db|storage_driver_host|storage_driver_password|storage_driver_secure|storage_driver_table|storage_driver_user|application-metrics-count-limit|boot-id-file|container-hints|containerd|docker-env-metadata-whitelist|docker-only|docker-root|docker-tls|docker-tls-ca|docker-tls-cert|docker-tls-key|enable-load-reader|event-storage-age-limit|event-storage-event-limit|global-housekeeping-interval|housekeeping-interval|log-cadvisor-usage|machine-id-file|storage-driver-buffer-duration|storage-driver-db|storage-driver-host|storage-driver-password|storage-driver-secure|storage-driver-table|storage-driver-user" ':(exclude)pkg/generated/bindata.go' ':(exclude)Godeps' ':(exclude)CHANGELOG*' ':(exclude)vendor'
```
- [x] --docker-root (**registered** - this is used in `cluster/saltbase/salt/kubelet/default`)
- [x] --housekeeping-interval (**registered** - e2e node tests rely on this (`test/e2e_node/resource_collector.go`))
- [x] --application-metrics-count-limit (**registered-deprecated** - temporarily register for legacy)
- [x] --boot-id-file (**registered-deprecated** - temporarily register for legacy)
- [x] --container-hints (**registered-deprecated** - temporarily register for legacy)
- [x] --containerd (**registered-deprecated** - temporarily register for legacy)
- [x] --docker (**registered-deprecated** - temporarily register for legacy)
- [x] --docker-env-metadata-whitelist (**registered-deprecated** - temporarily register for legacy)
- [x] --docker-only (**registered-deprecated** - temporarily register for legacy)
- [x] --docker-tls (**registered-deprecated** - temporarily register for legacy)
- [x] --docker-tls-ca (**registered-deprecated** - temporarily register for legacy)
- [x] --docker-tls-cert (**registered-deprecated** - temporarily register for legacy)
- [x] --docker-tls-key (**registered-deprecated** - temporarily register for legacy)
- [x] --enable-load-reader (**registered-deprecated** - temporarily register for legacy)
- [x] --event-storage-age-limit (**registered-deprecated** - the Kubelet overrides the default via the global flagset (`pkg/kubelet/cadvisor/cadvisor_linux.go`), but nothing else in core repo provides)
- [x] --event-storage-event-limit (**registered-deprecated** - the Kubelet overrides the default via the global flagset (`pkg/kubelet/cadvisor/cadvisor_linux.go`), but nothing else in core repo provides)
- [x] --global-housekeeping-interval (**registered-deprecated** - temporarily register for legacy)
- [x] --log-cadvisor-usage (**registered-deprecated** - temporarily register for legacy)
- [x] --machine-id-file (**registered-deprecated** - temporarily register for legacy)
- [x] --storage-driver-user (**registered-deprecated** - temporarily register for legacy)
- [x] --storage-driver-password (**registered-deprecated** - temporarily register for legacy)
- [x] --storage-driver-host (**registered-deprecated** - temporarily register for legacy)
- [x] --storage-driver-db (**registered-deprecated** - temporarily register for legacy)
- [x] --storage-driver-table (**registered-deprecated** - temporarily register for legacy)
- [x] --storage-driver-secure (**registered-deprecated** - temporarily register for legacy)
- [x] --storage-driver-buffer-duration (**registered-deprecated** - temporarily register for legacy)
### pkg/apiserver/util/logs:
- [x] --log-flush-frequency (**registered** - like the glog flags, this is probably useful)
### pkg/credentialprovider/azure/azure_credentials.go:
- [x] --azure-container-registry-config (**registered** - This isn't quite as straightforward as --google-json-key, because the file it points to isn't static. For now we will just register, and we will deprecate it when there is an alternative. See below comments.)
### pkg/credentialprovider/gcp/jwt.go:
- [x] --google-json-key (**registered-deprecated** - This is really old legacy stuff to allow kubelets to authenticate with gcr (see: d5e0054eb0). See @liggit's below comment for what should be used instead.)
### pkg/cloudprovider/providers/gce/gce_loadbalancer.go:
- [x] --cloud-provider-gce-lb-src-cidrs (**rejected** - Kubelet doesn't need to know about the cidrs that were opened in the firewall for the load balancer)
### glog:
I registered all of these, since this logging library is used pretty much everywhere in the Kubelet, and all of its toggles are probably useful.
- [x] --logtostderr (**registered**)
- [x] --alsologtostderr (**registered**)
- [x] -v, --v (**registered**)
- [x] --stderrthreshold (**registered**)
- [x] --vmodule (**registered**)
- [x] --log-backtrace-at (**registered**)
- [x] --log-dir (**registered**)
### verflag:
This is how you get the Kubelet's version, absolutely necessary to register this.
- [x] --version (**registered**)
```release-note
The Kubelet now explicitly registers all of its command-line flags with an internal flagset, which prevents flags from third party libraries from unintentionally leaking into the Kubelet's command-line API. Many unintentionally leaked flags are now marked deprecated, so that users have a chance to migrate away from them before they are removed. One previously leaked flag, --cloud-provider-gce-lb-src-cidrs, was entirely removed from the Kubelet's command-line API, because it is irrelevant to Kubelet operation.
```
Thanks to @liggitt for being surprised that we didn't already do this.
Thanks to @dashpole for realizing cadvisor flags were leaked after seeing #55863.
Thanks to @tallclair who recognized this problem a long time ago in #19432.
This explicitly registers Kubelet flags from libraries that were
registering flags globally, and stops parsing the global flag set.
In general, we should always be explicit about flags we register
and parse, so that we maintain control over our command-line API.
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Bump pause container used by kubelet and tests to 3.1
This updates the version of the pause container used by the kubelet and
various test utilities to 3.1.
**What this PR does / why we need it**: The pause container hasn't been rebuilt in quite a while and needs an update to reap zombies (#50865) and for schema2 manifest (#56253).
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#50865, Fixes#56253
**Special notes for your reviewer**:
**Release note**:
```release-note
The kubelet uses a new release 3.1 of the pause container with the Docker runtime. This version will clean up orphaned zombie processes that it inherits.
```
Automatic merge from submit-queue (batch tested with PRs 57746, 57621, 56839, 57464). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fix PodCIDR flag: defaults come from the object, not as literal args to the flag function
The defaulter runs on the object before adding flags. Flags should be registered with defaults sourced from this object, so that the defaulter, not the flag var function, determines the canonical default value.
```release-note
NONE
```
The first call to Set will clear the map before adding entries;
subsequent calls will simply append to the map.
This makes it possible to override default values with a command-line
option rather than appending to defaults,
while still allowing the distribution of key-value pairs across
multiple flag invocations.
For example: `--flag "a:hello" --flag "b:again" --flag "b:beautiful"
--flag "c:world"` results in `{"a": ["hello"], "b": ["again",
"beautiful"], "c": ["world"]}`
- Changes the following KubeletConfiguration fields from `string` to
`map[string]string`:
- `EvictionHard`
- `EvictionSoft`
- `EvictionSoftGracePeriod`
- `EvictionMinimumReclaim`
- Adds flag parsing shims to maintain Kubelet's public flags API, while
enabling structured input in the file API.
- Also removes `kubeletconfig.ConfigurationMap`, which was an ad-hoc flag
parsing shim living in the kubeletconfig API group, and replaces it
with the `MapStringString` shim introduced in this PR. Flag parsing
shims belong in a common place, not in the kubeletconfig API.
I manually audited these to ensure that this wouldn't cause errors
parsing the command line for syntax that would have previously been
error free (`kubeletconfig.ConfigurationMap` was unique in that it
allowed keys to be provided on the CLI without values. I believe this was
done in `flags.ConfigurationMap` to facilitate the `--node-labels` flag,
which rightfully accepts value-free keys, and that this shim was then
just copied to `kubeletconfig`). Fortunately, the affected fields
(`ExperimentalQOSReserved`, `SystemReserved`, and `KubeReserved`) expect
non-empty strings in the values of the map, and as a result passing the
empty string is already an error. Thus requiring keys shouldn't break
anyone's scripts.
- Updates code and tests accordingly.
Regarding eviction operators, directionality is already implicit in the
signal type (for a given signal, the decision to evict will be made when
crossing the threshold from either above or below, never both). There is
no need to expose an operator, such as `<`, in the API. By changing
`EvictionHard` and `EvictionSoft` to `map[string]string`, this PR
simplifies the experience of working with these fields via the
`KubeletConfiguration` type. Again, flags stay the same.
Other things:
- There is another flag parsing shim, `flags.ConfigurationMap`, from the
shared flag utility. The `NodeLabels` field still uses
`flags.ConfigurationMap`. This PR moves the allocation of the
`map[string]string` for the `NodeLabels` field from
`AddKubeletConfigFlags` to the defaulter for the external
`KubeletConfiguration` type. Flags are layered on top of an internal
object that has undergone conversion from a defaulted external object,
which means that previously the mere registration of flags would have
overwritten any previously-defined defaults for `NodeLabels` (fortunately
there were none).
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fix incorrect parameter tip
**What this PR does / why we need it**:
run kubelet set --init-config-dir=xxx, kubelet not work. see the error log need to open KubeletConfigFile feature gates.
But
kubelet --help
--init-config-dir string The Kubelet will look in this directory for the init configuration. The path may be absolute or relative; relative paths start at the Kubelet's current working directory. Omit this argument to use the built-in default configuration values. Presently, you must also enable the `DynamicKubeletConfig` feature gate to pass this flag.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes ##55666
**Special notes for your reviewer**:
**Release note**:
```
NONE
```
Automatic merge from submit-queue (batch tested with PRs 52367, 53363, 54989, 54872, 54643). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Lift embedded structure out of ManifestURLHeader field
Related: #53833
```release-note
It is now possible to set multiple manifest url headers via the Kubelet's --manifest-url-header flag. Multiple headers for the same key will be added in the order provided. The ManifestURLHeader field in KubeletConfiguration object (kubeletconfig/v1alpha1) is now a map[string][]string, which facilitates writing JSON and YAML files.
```
