During volume detach, the following might happen in reconciler
1. Pod is deleting
2. remove volume from reportedAsAttached, so node status updater will
update volumeAttached list
3. detach failed due to some issue
4. volume is added back in reportedAsAttached
5. reconciler loops again the volume, remove volume from
reportedAsAttached
6. detach will not be trigged because exponential back off, detach call
will fail with exponential backoff error
7. another pod is added which using the same volume on the same node
8. reconciler loops and it will NOT try to tigger detach anymore
At this point, volume is still attached and in actual state, but
volumeAttached list in node status does not has this volume anymore, and
will block volume mount from kubelet.
The fix in first round is to add volume back into the volume list that
need to reported as attached at step 6 when detach call failed with
error (exponentical backoff). However this might has some performance
issue if detach fail for a while. During this time, volume will be keep
removing/adding back to node status which will cause a surge of API
calls.
So we changed to logic to check first whether operation is safe to retry which
means no pending operation or it is not in exponentical backoff time
period before calling detach. This way we can avoid keep removing/adding
volume from node status.
Change-Id: I5d4e760c880d72937d34b9d3e904ecad125f802e
There is no reason for having separate owners for this folder. The parent
folder has a much better OWNERS file with references to the SIG-Storage
aliases.
This cleans up a log message that looks like:
I0312 14:36:50.280018 12866 operation_generator.go:869] UnmountDevice succeeded for volume "my-volume" %!(EXTRA string=UnmountDevice succeeded for volume "my-volume" (UniqueName: "kubernetes.io/csi/smb.csi.k8s.io^my-volume") on node "my-node")
Prevent Kubelet from incorrectly interpreting "not yet started" pods as "ready to terminate pods" by unifying responsibility for pod lifecycle into pod worker
should mark volume mount in actual state even if volume expansion fails so that
reconciler can tear down the volume when needed. To avoid pods start
using it, mark volume as uncertain instead of mounted.
Will add unit test after the logic is reviewed.
Change-Id: I5aebfa11ec93235a87af8f17bea7f7b1570b603d
A number of race conditions exist when pods are terminated early in
their lifecycle because components in the kubelet need to know "no
running containers" or "containers can't be started from now on" but
were relying on outdated state.
Only the pod worker knows whether containers are being started for
a given pod, which is required to know when a pod is "terminated"
(no running containers, none coming). Move that responsibility and
podKiller function into the pod workers, and have everything that
was killing the pod go into the UpdatePod loop. Split syncPod into
three phases - setup, terminate containers, and cleanup pod - and
have transitions between those methods be visible to other
components. After this change, to kill a pod you tell the pod worker
to UpdatePod({UpdateType: SyncPodKill, Pod: pod}).
Several places in the kubelet were incorrect about whether they
were handling terminating (should stop running, might have
containers) or terminated (no running containers) pods. The pod worker
exposes methods that allow other loops to know when to set up or tear
down resources based on the state of the pod - these methods remove
the possibility of race conditions by ensuring a single component is
responsible for knowing each pod's allowed state and other components
simply delegate to checking whether they are in the window by UID.
Removing containers now no longer blocks final pod deletion in the
API server and are handled as background cleanup. Node shutdown
no longer marks pods as failed as they can be restarted in the
next step.
See https://docs.google.com/document/d/1Pic5TPntdJnYfIpBeZndDelM-AbS4FN9H2GTLFhoJ04/edit# for details
When UnmountDevice fails, kubelet treat the volume mount as uncertain,
because it does not know at which stage UnmountDevice failed. It may be
already partially unmonted / destroyed.
As result, MountDevice will be performer when a new Pod is started on the
node after UnmountDevice faiure.
iSCSI and FC volume plugins do not implement real 3rd party attach/detach.
If reconstruction fails with an error on a FC or iSCSI volume, it will not
be unmounted from the volume global dir and at the same time it will be
marked as unused, to be available to be mounted on another node.
The volume can then be mounted on several nodes, resulting in volume
corruption.
The other block based volume plugins implement attach/detach that either
makes the volume stuck (can't be detached) or will be force-detached from a
node before attaching it somewhere else.
The non "util-linux" versions of "losetup" don't seem to have
options like "-j" ("--associated") which lists the loop devices
associated with the file, or "--show" which displays the name
of the assigned loop device for a file.
For instance, when "-j" is used, "GetLoopDevice()" fails with:
$ losetup -j /path/to/file
losetup: unrecognized option: j
BusyBox v1.32.1 () multi-call binary.
Add a fallback option to lookup the device from "sysfs" in cases
where "losetup" fails for an invalid option. This can be done by
reading the backing file from "/sys/block/loop*/loop/backing_file"
for each of the devices listed there.
Signed-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>
Fix inode usage calculation to use filepath.Walk instead of executing an
external find. Also calculate the disk usage while at it so we also get
rid of the external dependency of `nice` and `du`. (#95172)
This is similar to what cadvisor does since commit
046818d64c
This solves three problems:
- Counts number of inodes correct when there are hardlinks (#96114)
- Makes kubelet work without GNU findutils (#95186)
- Makes kubelet work without GNU coreutils (#95172)
