Kubelet restart process seems to get a bit slower recently. From running the gpu_device_plugin e2e_node test on GCE, I saw it took ~37 seconds for kubelet to start CM DeviceManager after it restarts, and then took ~12 seconds for the gpu device plugin to re-register. As the result, this e2e_node test fails because the current 10 sec waiting time is too small. Restarting a container also seems to get slower that it sometimes exceeds the current 2 min waiting time in ensurePodContainerRestart(). This change increase both waiting time to 5 min to leave enough space on slower machines.
6.5 KiB
6.5 KiB