Automatic merge from submit-queue (batch tested with PRs 45877, 46846, 46630, 46087, 47003)
gpusInUse info error when kubelet restarts
**What this PR does / why we need it**:
In my test, I found 2 errors in the nvidia_gpu_manager.go.
1. the number of activePods in gpusInUse() equals to 0 when kubelet restarts. It seems the Start() method was called before pods recovery which caused this error. So I decide not to call gpusInUse() in the Start() function, just let it happen when new pod needs to be created.
2. the container.ContainerID in line 242 returns the id in format of "docker://<container_id>", this will make the client failed to inspect the container by id. We have to erase the prefix of "docker://".
**Special notes for your reviewer**:
**Release note**:
```
Avoid assigning the same GPU to multiple containers.
```
Automatic merge from submit-queue (batch tested with PRs 45877, 46846, 46630, 46087, 47003)
func parseEndpointWithFallbackProtocol should check if protocol of endpoint is empty
**What this PR does / why we need it**:
func parseEndpointWithFallbackProtocol should check if protocol of endpoint is empty
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: part of #45927
NONE
**Special notes for your reviewer**:
NONE
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 43005, 46660, 46385, 46991, 47103)
Consolidate sysctl commands for kubelet
**What this PR does / why we need it**:
These commands are important enough to be in the Kubelet itself.
By default, Ubuntu 14.04 and Debian Jessie have these set to 200 and
20000. Without this setting, nodes are limited in the number of
containers that they can start.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#26005
**Special notes for your reviewer**:
I had a difficult time writing tests for this. It is trivial to create a fake sysctl for testing, but the Kubelet does not have any tests for the prior settings.
**Release note**:
```release-note
```
Automatic merge from submit-queue (batch tested with PRs 46775, 47009)
kuberuntime: check the value of RunAsNonRoot when verifying
The verification function is fixed to check the value of RunAsNonRoot,
not just the existence of it. Also adds unit tests to verify the correct
behavior.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#46996
**Release note**:
```release-note
Fix the bug where container cannot run as root when SecurityContext.RunAsNonRoot is false.
```
The verification function is fixed to check the value of RunAsNonRoot,
not just the existence of it. Also adds unit tests to verify the correct
behavior.
This PR adds two features:
1. add support for isolating the emptyDir volume use. If user
sets a size limit for emptyDir volume, kubelet's eviction manager
monitors its usage
and evict the pod if the usage exceeds the limit.
2. add support for isolating the local storage for container overlay. If
the container's overly usage exceeds the limit defined in container
spec, eviction manager will evict the pod.
Automatic merge from submit-queue (batch tested with PRs 46734, 46810, 46759, 46259, 46771)
Improve code coverage for pkg/kubelet/images/image_gc_manager
**What this PR does / why we need it**:
#39559#40780
code coverage from 74.5% to 77.4%
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue
Delete all dead containers and sandboxes when under disk pressure.
This PR modifies the eviction manager to add dead container and sandbox garbage collection as a resource reclaim function for disk. It also modifies the container GC logic to allow pods that are terminated, but not deleted to be removed.
It still does not delete containers that are less than the minGcAge. This should prevent nodes from entering a permanently bad state if the entire disk is occupied by pods that are terminated (in the state failed, or succeeded), but not deleted.
There are two improvements we should consider making in the future:
- Track the disk space and inodes reclaimed by deleting containers. We currently do not track this, and it prevents us from determining if deleting containers resolves disk pressure. So we may still evict a pod even if we are able to free disk space by deleting dead containers.
- Once we can track disk space and inodes reclaimed, we should consider only deleting the containers we need to in order to relieve disk pressure. This should help avoid a scenario where we try and delete a massive number of containers all at once, and overwhelm the runtime.
/assign @vishh
cc @derekwaynecarr
```release-note
Disk Pressure triggers the deletion of terminated containers on the node.
```
Automatic merge from submit-queue
reset resultRun on pod restart
xref https://bugzilla.redhat.com/show_bug.cgi?id=1455056
There is currently an issue where, if the pod is restarted due to liveness probe failures exceeding failureThreshold, the failure count is not reset on the probe worker. When the pod restarts, if the liveness probe fails even once, the pod is restarted again, not honoring failureThreshold on the restart.
```yaml
apiVersion: v1
kind: Pod
metadata:
name: busybox
spec:
containers:
- name: busybox
image: busybox
command:
- sleep
- "3600"
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 3
timeoutSeconds: 1
periodSeconds: 3
successThreshold: 1
failureThreshold: 5
terminationGracePeriodSeconds: 0
```
Before this PR:
```
$ kubectl create -f busybox-probe-fail.yaml
pod "busybox" created
$ kubectl get pod -w
NAME READY STATUS RESTARTS AGE
busybox 1/1 Running 0 4s
busybox 1/1 Running 1 24s
busybox 1/1 Running 2 33s
busybox 0/1 CrashLoopBackOff 2 39s
```
After this PR:
```
$ kubectl create -f busybox-probe-fail.yaml
$ kubectl get pod -w
NAME READY STATUS RESTARTS AGE
busybox 0/1 ContainerCreating 0 2s
busybox 1/1 Running 0 4s
busybox 1/1 Running 1 27s
busybox 1/1 Running 2 45s
```
```release-note
Fix kubelet reset liveness probe failure count across pod restart boundaries
```
Restarts are now happen at even intervals.
@derekwaynecarr
Automatic merge from submit-queue (batch tested with PRs 46782, 46719, 46339, 46609, 46494)
Do not log the content of pod manifest if parsing fails.
**What this PR does / why we need it**:
- ~~only accepts text/plain config file~~
- ~~not log config file content when it's invalid~~
Do not log the content of pod manifest if parsing fails.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#46493
**Special notes for your reviewer**:
/cc @yujuhong
@sig-node-reviewers
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 46782, 46719, 46339, 46609, 46494)
Fix inconsistency in finding cni binaries
Fixes [#46476]
Signed-off-by: Abhinav Dahiya <abhinav.dahiya@coreos.com>
**What this PR does / why we need it**:
This fixes the inconsistency in finding the appropriate cni binaries.
Currently `lo` cniNetwork follows vendorCniDir > binDir whereas default for all others is binDir > vendorCniDir. This PR makes vendorCniDir > binDir as default behavior.
**Why we need it**:
Hypercube right now ships cni binaries in /opt/cni/bin.
And to use latest version of calico you need to override kubelet's /opt/cni/bin from host which means all other cni plugins (flannel, loopback etc...) have to be mounted from host too. Keeping vendordir at higher order allows easy installation of newer versions of plugins.
Automatic merge from submit-queue (batch tested with PRs 46620, 46732, 46773, 46772, 46725)
Improving test coverage for kubelet/kuberuntime.
**What this PR does / why we need it**:
Increases test coverage for kubelet/kuberuntime
https://github.com/kubernetes/kubernetes/issues/46123
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
https://github.com/kubernetes/kubernetes/issues/46123
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue
Add local storage (scratch space) allocatable support
This PR adds the support for allocatable local storage (scratch space).
This feature is only for root file system which is shared by kubernetes
componenets, users' containers and/or images. User could use
--kube-reserved flag to reserve the storage for kube system components.
If the allocatable storage for user's pods is used up, some pods will be
evicted to free the storage resource.
This feature is part of local storage capacity isolation and described in the proposal https://github.com/kubernetes/community/pull/306
**Release note**:
```release-note
This feature exposes local storage capacity for the primary partitions, and supports & enforces storage reservation in Node Allocatable
```
Automatic merge from submit-queue (batch tested with PRs 46239, 46627, 46346, 46388, 46524)
move labels to components which own the APIs
During the apimachinery split in 1.6, we accidentally moved several label APIs into apimachinery. They don't belong there, since the individual APIs are not general machinery concerns, but instead are the concern of particular components: most commonly the kubelet. This pull moves the labels into their owning components and out of API machinery.
@kubernetes/sig-api-machinery-misc @kubernetes/api-reviewers @kubernetes/api-approvers
@derekwaynecarr since most of these are related to the kubelet
Automatic merge from submit-queue (batch tested with PRs 46726, 41912, 46695, 46034, 46551)
Rotate kubelet client certificate.
Changes the kubelet so it bootstraps off the cert/key specified in the
config file and uses those to request new cert/key pairs from the
Certificate Signing Request API, as well as rotating client certificates
when they approach expiration.
Default behavior is for client certificate rotation to be disabled. If enabled
using a command line flag, the kubelet exits each time the certificate is
rotated. I tried to use `GetCertificate` in [tls.Config](https://golang.org/pkg/crypto/tls/#Config) but it is only called
on the server side of connections. Then I tried `GetClientCertificate`,
but it is new in 1.8.
**Release note**
```release-note
With --feature-gates=RotateKubeletClientCertificate=true set, the kubelet will
request a client certificate from the API server during the boot cycle and pause
waiting for the request to be satisfied. It will continually refresh the certificate
as the certificates expiration approaches.
```
Automatic merge from submit-queue
Improved code coverage for pkg/kubelet/util.
The test coverage for pkg/kubelet/util.go increased from 45.1%
to 84.3%.
**What this PR does / why we need it**:
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
```
Automatic merge from submit-queue (batch tested with PRs 46432, 46701, 46326, 40848, 46396)
Fix selfLinks of pods started from manifests
**What this PR does / why we need it**:
When running `curl http://localhost:10255/pods` the selfLink for pods started from manifests were incorrect. This PR fixes it.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#46357
**Special notes for your reviewer**:
@number101010
**Release note**:
```release-note
NONE
```
This PR adds the check for local storage request when admitting pods. If
the local storage request exceeds the available resource, pod will be
rejected.
This PR adds the support for allocatable local storage (scratch space).
This feature is only for root file system which is shared by kubernetes
componenets, users' containers and/or images. User could use
--kube-reserved flag to reserve the storage for kube system components.
If the allocatable storage for user's pods is used up, some pods will be
evicted to free the storage resource.
Automatic merge from submit-queue
fix comment error in function newVolumeMounterFromPlugins
**What this PR does / why we need it**:
Fix the comment error in function newVolumeMounterFromPlugins, which may cause confusion.
Automatic merge from submit-queue
resolv.conf nameserver line has only one entry, ignore trailing garbage
**What this PR does / why we need it**:
Per the resolv.conf man page "name servers may be listed, one per keyword." Some tools such as udhcpc take advantage of this to append comments to nameserver entries. For example: `nameserver 8.8.8.8 # eth0`. This updates the resolv.conf parser to ignore trailing garbage on nameserver lines.
**Release note**:
NONE
Changes the kubelet so it bootstraps off the cert/key specified in the
config file and uses those to request new cert/key pairs from the
Certificate Signing Request API, as well as rotating client certificates
when they approach expiration.
Automatic merge from submit-queue (batch tested with PRs 46661, 46562, 46657, 46655, 46640)
remove redundant carriage return for readable
**What this PR does / why we need it**:
remove redundant carriage to make it more readable.
Automatic merge from submit-queue (batch tested with PRs 46076, 43879, 44897, 46556, 46654)
kubelet status manager: short-circut the evaluation if possible
This avoids redundant checks in the `PodResourcesAreReclaimed`, which
also logs misleading "Pod is terminated" messages on every check
/cc @kubernetes/sig-node-pr-reviews
Automatic merge from submit-queue (batch tested with PRs 46076, 43879, 44897, 46556, 46654)
Local storage plugin
**What this PR does / why we need it**:
Volume plugin implementation for local persistent volumes. Scheduler predicate will direct already-bound PVCs to the node that the local PV is at. PVC binding still happens independently.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*:
Part of #43640
**Release note**:
```
Alpha feature: Local volume plugin allows local directories to be created and consumed as a Persistent Volume. These volumes have node affinity and pods will only be scheduled to the node that the volume is at.
```
Automatic merge from submit-queue (batch tested with PRs 46076, 43879, 44897, 46556, 46654)
kubelet/network: report but tolerate errors returned from GetNetNS()
Runtimes should never return "" and nil errors, since network plugin
drivers need to treat netns differently in different cases. So return
errors when we can't get the netns, and fix up the plugins to do the
right thing.
Namely, we don't need a NetNS on pod network teardown. We do need
a netns for pod Status checks and for network setup.
@kubernetes/rh-networking @kubernetes/sig-network-bugs @DirectXMan12
Automatic merge from submit-queue (batch tested with PRs 46635, 45619, 46637, 45059, 46415)
Certificate rotation for kubelet server certs.
Replaces the current kubelet server side self signed certs with certs signed by
the Certificate Request Signing API on the API server. Also renews expiring
kubelet server certs as expiration approaches.
Two Points:
1. With `--feature-gates=RotateKubeletServerCertificate=true` set, the kubelet will
request a certificate during the boot cycle and pause waiting for the request to
be satisfied.
2. In order to have the kubelet's certificate signing request auto approved,
`--insecure-experimental-approve-all-kubelet-csrs-for-group=` must be set on
the cluster controller manager. There is an improved mechanism for auto
approval [proposed](https://github.com/kubernetes/kubernetes/issues/45030).
**Release note**:
```release-note
With `--feature-gates=RotateKubeletServerCertificate=true` set, the kubelet will
request a server certificate from the API server during the boot cycle and pause
waiting for the request to be satisfied. It will continually refresh the certificate as
the certificates expiration approaches.
```
Automatic merge from submit-queue (batch tested with PRs 44061, 46614, 46419, 46628, 46134)
cleanup kubelet new node status test
**What this PR does / why we need it**:
this scopes the test to just testing allocatable values. extra parts of the original test were copied from another test that was not relevant.
Automatic merge from submit-queue
kubelet: group all container-runtime-specific flags/options into a separate struct
They don't belong in the KubeletConfig.
This addresses #43253
Automatic merge from submit-queue
add myself and liggitt to pkg/kubelet/certificats OWNERs
For as long a kubelet is using the internal client, this certificate
manager is bound to the kubelet. Once kubelet has moved to client-go we
plan to extract this library to be general purpose. In the meantime,
liggitt and I should handle reviews of this code.
@liggitt @timstclair
