kubernetes/pkg/kubelet/cm/devicemanager
Francesco Romani 2f426fdba6 devicemanager: checkpoint: support pre-1.20 data
The commit a8b8995ef2
changed the content of the data kubelet writes in the checkpoint.
Unfortunately, the checkpoint restore code was not updated,
so if we upgrade kubelet from pre-1.20 to 1.20+, the
device manager cannot anymore restore its state correctly.

The only trace of this misbehaviour is this line in the
kubelet logs:
```
W0615 07:31:49.744770    4852 manager.go:244] Continue after failing to read checkpoint file. Device allocation info may NOT be up-to-date. Err: json: cannot unmarshal array into Go struct field PodDevicesEntry.Data.PodDeviceEntries.DeviceIDs of type checkpoint.DevicesPerNUMA
```

If we hit this bug, the device allocation info is
indeed NOT up-to-date up until the device plugins register
themselves again. This can take up to few minutes, depending
on the specific device plugin.

While the device manager state is inconsistent:
1. the kubelet will NOT update the device availability to zero, so
   the scheduler will send pods towards the inconsistent kubelet.
2. at pod admission time, the device manager allocation will not
   trigger, so pods will be admitted without devices actually
   being allocated to them.

To fix these issues, we add support to the device manager to
read pre-1.20 checkpoint data. We retroactively call this
format "v1".

Signed-off-by: Francesco Romani <fromani@redhat.com>
2021-10-26 09:54:11 +02:00
..
checkpoint devicemanager: checkpoint: support pre-1.20 data 2021-10-26 09:54:11 +02:00
device_plugin_stub.go Migrate missed log entries in kubelet 2021-03-18 14:26:26 -07:00
endpoint_test.go Use unique socket name per cm test 2020-09-04 14:55:23 -04:00
endpoint.go add info for possible failed listAndWatch grpc call 2021-06-17 16:25:20 +08:00
manager_stub.go node: podresources: make GetDevices() consistent 2021-03-09 13:13:36 +01:00
manager_test.go devicemanager: checkpoint: support pre-1.20 data 2021-10-26 09:54:11 +02:00
manager.go devicemanager: checkpoint: support pre-1.20 data 2021-10-26 09:54:11 +02:00
OWNERS Add klueska as reviewer for CPUManager and devicemanager 2019-10-03 13:01:41 -07:00
pod_devices_test.go devicemanager: skip unhealthy devs in GetAllocatable 2021-09-22 19:20:04 +02:00
pod_devices.go Merge pull request #103289 from AlexeyPerevalov/DoNotExportEmptyTopology 2021-10-07 07:11:46 -07:00
topology_hints_test.go node: podresources: make GetDevices() consistent 2021-03-09 13:13:36 +01:00
topology_hints.go device manager: do not clean admitted pods from the state 2021-08-08 16:46:06 +03:00
types.go Migrate devicemanager to structured logging 2021-03-14 11:57:06 -04:00