github/kubernetes
4
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
87e32c7532
kubernetes/pkg/kubelet
History
Kubernetes Submit Queue 6de28fab7d Merge pull request #42942 from vishh/gpu-cont-fix 
Automatic merge from submit-queue (batch tested with PRs 42942, 42935)

[Bug] Handle container restarts and avoid using runtime pod cache while allocating GPUs

Fixes #42412

**Background**
Support for multiple GPUs is an experimental feature in v1.6. 
Container restarts were handled incorrectly which resulted in stranding of GPUs
Kubelet is incorrectly using runtime cache to track running pods which can result in race conditions (as it did in other parts of kubelet). This can result in same GPU being assigned to multiple pods.

**What does this PR do**
This PR tracks assignment of GPUs to containers and returns pre-allocated GPUs instead of (incorrectly) allocating new GPUs.
GPU manager is updated to consume a list of active pods derived from apiserver cache instead of runtime cache.
Node e2e has been extended to validate this failure scenario.

**Risk**
Minimal/None since support for GPUs is an experimental feature that is turned off by default. The code is also isolated to GPU manager in kubelet.

**Workarounds**
In the absence of this PR, users can mitigate the original issue by setting `RestartPolicyNever`  in their pods.
There is no workaround for the race condition caused by using the runtime cache though.
Hence it is worth including this fix in v1.6.0.

cc @jianzhangbjz @seelam @kubernetes/sig-node-pr-reviews 

Replaces #42560
2017-03-14 10:19:17 -07:00
..
api add volume timestamps 2017-03-02 15:01:59 -08:00
cadvisor Update generated files 2017-01-29 21:41:45 +01:00
certificate Rotate the kubelet certificate when about to expire. 2017-02-17 17:42:35 -08:00
client find and replace 2017-01-20 08:04:53 -05:00
cm Fix kubelet panic in cgroup manager. 2017-03-13 12:06:08 -07:00
config Ignore file start with dots 2017-02-20 21:49:42 +08:00
container kubelet: add cgroup manager metrics 2017-03-06 08:53:47 -06:00
custommetrics Enable auto-generating sources rules 2017-01-05 14:14:13 -08:00
dockershim Merge pull request #42739 from yujuhong/created_time 2017-03-09 02:51:38 -08:00
dockertools Merge pull request #42694 from dchen1107/master 2017-03-09 15:07:00 -08:00
envvars refactor: use metav1.ObjectMeta in other types 2017-01-17 16:17:19 -05:00
events Implement support for mount options in PVs 2017-03-01 11:50:40 -05:00
eviction Merge pull request #42435 from dashpole/timestamps_for_fsstats 2017-03-03 23:21:48 -08:00
gpu remove dead code in gpu manager 2017-03-13 10:58:26 -07:00
images switch kubelet to use external (client-go) object references for events 2017-01-31 19:15:33 -05:00
kuberuntime Add more logs to help debugging 2017-03-08 12:27:49 -08:00
leaky Enable auto-generating sources rules 2017-01-05 14:14:13 -08:00
lifecycle critical pods can preempt other pods to be admitted 2017-02-23 10:31:20 -08:00
metrics kubelet: add cgroup manager metrics 2017-03-06 08:53:47 -06:00
network teach kubenet to use annotation instead of pod object for traffic shaper 2017-02-27 10:11:09 -08:00
pleg move authoritative client-go utils out of pkg 2017-01-24 08:59:18 -05:00
pod refactor: Move *Options references to metav1 2017-01-24 13:41:51 -05:00
preemption critical pods can preempt other pods to be admitted 2017-02-23 10:31:20 -08:00
prober ExecProbes should be able to do simple env var substitution 2017-02-28 22:46:04 -05:00
qos Avoid evicting critical pods in Kubelet if a special feature gate is enabled 2017-02-02 11:32:20 -08:00
remote Fix typo of defualt 2017-02-11 22:28:24 +08:00
rkt Added ability for Docker containers to set usage of dns settings along with hostNetwork is true 2017-03-01 17:10:00 +00:00
rktshim Merge pull request #39158 from feiskyer/cri-proto3 2017-01-20 08:28:49 -08:00
secret Use consistent helper for getting secret names from pod 2017-02-23 00:40:17 -05:00
server Add pprof trace support 2017-03-07 10:10:42 -05:00
status do not skip pods that can be deleted 2017-03-09 09:35:50 -08:00
sysctl Enable auto-generating sources rules 2017-01-05 14:14:13 -08:00
types Avoid evicting critical pods in Kubelet if a special feature gate is enabled 2017-02-02 11:32:20 -08:00
util Split RequestNodeCertificate function. 2017-02-17 07:40:48 -08:00
volumemanager Merge pull request #41456 from dashpole/pod_volume_cleanup 2017-02-16 10:14:05 -08:00
active_deadline_test.go move client/record 2017-01-31 19:14:13 -05:00
active_deadline.go move client/record 2017-01-31 19:14:13 -05:00
BUILD adding an e2e for GPUs 2017-02-28 13:42:08 -08:00
disk_manager_test.go Revert "Declare out of disk when there is no free inodes" 2016-07-06 08:19:09 -07:00
disk_manager.go Revert "Declare out of disk when there is no free inodes" 2016-07-06 08:19:09 -07:00
doc.go Use Go canonical import paths 2016-07-16 13:48:21 -04:00
kubelet_cadvisor_test.go kubelet: Make cadvisor GetContainerInfo tests table driven 2017-02-15 16:15:21 +01:00
kubelet_cadvisor.go start the apimachinery repo 2017-01-11 09:09:48 -05:00
kubelet_getters_test.go kubelet: Fix ConfigMap on Windows. 2017-01-24 18:40:49 +00:00
kubelet_getters.go kubelet: Fix ConfigMap on Windows. 2017-01-24 18:40:49 +00:00
kubelet_network_test.go move client/record 2017-01-31 19:14:13 -05:00
kubelet_network.go Fixed forming of pod's Search line in resolv.conf: 2017-01-17 13:18:26 +00:00
kubelet_node_status_test.go fix a bug in nvidia gpu allocation and added unit test 2017-02-28 13:42:08 -08:00
kubelet_node_status.go Revert "Revert "Pods pending due to insufficient OIR should get scheduled once sufficient OIR becomes available."" 2017-03-06 17:48:09 -08:00
kubelet_pods_test.go Invalid environment var names are reported and pod starts 2017-03-09 07:21:53 +00:00
kubelet_pods_windows_test.go kubelet: Fix ConfigMap on Windows. 2017-01-24 18:40:49 +00:00
kubelet_pods.go use active pods instead of runtime pods in gpu manager 2017-03-13 10:58:26 -07:00
kubelet_resources_test.go add support for node allocatable phase 2 to kubelet 2017-02-27 21:24:44 -08:00
kubelet_resources.go add support for node allocatable phase 2 to kubelet 2017-02-27 21:24:44 -08:00
kubelet_test.go improve gpu integration 2017-02-28 11:27:53 -08:00
kubelet_volumes_test.go move testing/core to client-go 2017-01-26 13:54:40 -05:00
kubelet_volumes.go kubelet volumes cleanupOrphanedPodDirs does not distinguish error from found volume paths 2017-02-17 09:07:54 -05:00
kubelet.go use active pods instead of runtime pods in gpu manager 2017-03-13 10:58:26 -07:00
networks.go add portmapping getter into network host 2017-02-08 09:35:04 -08:00
oom_watcher_test.go switch kubelet to use external (client-go) object references for events 2017-01-31 19:15:33 -05:00
oom_watcher.go switch kubelet to use external (client-go) object references for events 2017-01-31 19:15:33 -05:00
OWNERS Small fix in kubelet owner file. 2017-01-19 11:04:57 -08:00
pod_container_deletor_test.go Delete all dead containers only after pod syncing is done. 2016-08-15 14:36:51 -07:00
pod_container_deletor.go start the apimachinery repo 2017-01-11 09:09:48 -05:00
pod_workers_test.go move client/record 2017-01-31 19:14:13 -05:00
pod_workers.go increase timeout 2017-02-27 10:59:24 -08:00
reason_cache_test.go start the apimachinery repo 2017-01-11 09:09:48 -05:00
reason_cache.go start the apimachinery repo 2017-01-11 09:09:48 -05:00
runonce_test.go Revert "Merge pull request #41202 from dashpole/revert-41095-deletion_pod_lifecycle" 2017-02-15 08:44:03 -08:00
runonce.go cmd/kubelet 2016-11-23 15:53:09 -08:00
runtime.go Check the health of PLEG when updating the node status 2017-01-10 16:34:00 -08:00
util.go cmd/kubelet 2016-11-23 15:53:09 -08:00
volume_host.go SecretVolume using secret manager 2017-01-23 16:10:01 +01:00