Commit Graph

415 Commits

Author SHA1 Message Date
Jan Safranek
9ca548fcf0 Add metrics for force cleaned mounts after failed reconstruction
Count nr. of force cleaned mounts + their failures after a volume fails
reconstruction.
2023-03-06 17:48:59 +01:00
Jan Safranek
bd73aee9db Add volume reconstruction metrics
Count nr. of volumes that kubelet tried to reconstruct + reconstruction
errors.
2023-02-22 13:01:26 +01:00
Kubernetes Prow Robot
5071c4f57e
Merge pull request #111982 from cvvz/kubelet-del-unnecessary-code
cleanup: delete useless code from kubelet volumemanager
2023-02-14 10:31:31 -08:00
weizhi
575616cc72
fix: kubelet event about unattached volumes is incorrect (#112719)
* fix kubelet event

* add ut

* fix ut
2023-01-04 01:51:59 -08:00
Kubernetes Prow Robot
6a48538090
Merge pull request #114326 from sunnylovestiramisu/fixReconstructionBug
Add pod to dsw if termination is not completed during reconstruction
2022-12-20 16:31:24 -08:00
Jordan Liggitt
78cb3862f1
Fix indentation/spacing in comments to render correctly in godoc 2022-12-17 23:27:38 -05:00
Sunny Song
5e8ca18308 Add pod to dsw if termination is not completed during reconstruction #issues/113979 2022-12-16 16:32:34 -08:00
Kubernetes Prow Robot
a668924cb6
Merge pull request #113255 from claudiubelu/path-filepath-update-kubelet
Replaces path.Operation with filepath.Operation (kubelet)
2022-12-09 22:27:41 -08:00
arrowfeng
6a57404e28 kubelet: cleanup secretManager and configManager in podManager
Signed-off-by: arrowfeng <289716347@qq.com>
2022-11-14 23:05:32 +08:00
Claudiu Belu
b9bf3e5c49 Replaces path.Operation with filepath.Operation (kubelet)
The path module has a few different functions:
Clean, Split, Join, Ext, Dir, Base, IsAbs. These functions do not
take into account the OS-specific path separator, meaning that they
won't behave as intended on Windows.

For example, Dir is supposed to return all but the last element of the
path. For the path "C:\some\dir\somewhere", it is supposed to return
"C:\some\dir\", however, it returns ".".

Instead of these functions, the ones in filepath should be used instead.
2022-11-08 16:05:48 +00:00
Jan Safranek
cf912a2512 Update SELinux context log
SELinux context discovered from Pod is not final, it can be cleared when a
volume plugin does not support SELinux or the volume is not
ReadWriteOncePod. Update the existing log line + add a new one for easier
debugging.
2022-11-08 11:17:38 +01:00
Jan Safranek
e575e60ea4 Reconstruct SELinux mount option
When reconstructing volumes from disk after kubelet restart, reconstruct
also context=XYZ mount option and add it to the ActualStateOfWorld.
2022-11-08 11:17:38 +01:00
Jan Safranek
9a98f7318b Increase verbosity of volume reconstruction messages
Add volume reconstruction logs to V(2) to see initial kubelet
ActualStateOfWorld after kubelet start. Kubelet logs SetUp / TearDown
events at V(2) already, so we can track the whole volume mount state in
V(2) logs.
2022-11-07 11:05:27 +01:00
Jan Safranek
286e19c460 Add node name parameter
Add nodeName to MarkVolumeAsAttached. MarkVolumeAsAttached implementation
in kubelet does not use the parameter, but it could do that in the future.
2022-11-07 10:50:23 +01:00
Jan Safranek
20c5cc0a39 Add unit test for failed mount after reconstruction
To preserve fix in https://github.com/kubernetes/kubernetes/pull/110670,
add an unit test that check a volume is *uncertain* even after final mount
error when it was reconstructed.

And actually fix a regression introduced in the previous patch.
2022-11-04 12:25:21 +01:00
Jan Safranek
6d810f2cd4 Add unit tests 2022-11-03 17:54:58 +01:00
Jan Safranek
3a79466ddd Reshuffle functions between reconstruct and reconstruc_common
Move common functions to reconstruc_common.go and functions used only for
the current (old) reconstruction to reconstruct.go
2022-11-03 16:55:13 +01:00
Jan Safranek
44b72d0348 Move new reconciler logic into reconciler_new.go
Move reconciler logic from reconstruct{new}.go to:
- reconciler.go - only the functionality used by the current (old)
  reconciler.
- reconciler_new.go - only the functionality used by the new reconciler.
- reconciler_common.go - common functions.
2022-11-03 16:55:13 +01:00
Jan Safranek
fc245b339b Refactor ConstructVolumeSpec
Return a struct from ConstructVolumeSpec to be able to add more fields to
it later.
2022-11-03 16:55:13 +01:00
Jan Safranek
2dc8cc13a4 Remove SyncReconstructedVolume call
With the new reconstruction, AWS.MarkVolumeAsMounted will update outer spec
name with the correct value from Pod.
2022-11-03 16:55:12 +01:00
Jan Safranek
e0f3e5c457 Rework volume reconstruction
Subsequent SELinux work (see http://kep.k8s.io/1710) will need
ActualStateOfWorld populated around the time kubelet starts mounting
volumes.

Therefore reconstruct volumes before starting reconciler, but do not depend
on the desired state of world populated nor node.status - both need a
working API server, which may not be available at that time.

All reconstructed volumes are marked as Uncertain and reconciler will sort
them out - call SetUp to ensure the volume is really mounted when a pod
needs the volume or call TearDown then there is no such pod.

Finish the reconstruction when the API server becomes available:
- Clean up volumes that failed reconstruction and are not needed.

- Update devicePath of reconstructed volumes from node.status. Make sure
  not to overwrite devicePath that may have been updated when the volume
  was mounted by reconcile().

Hiding all this rework behind SELinuxMountReadWriteOncePod FeatureGate,
just to make sure we have a way back if this commit is buggy.
2022-11-03 16:55:12 +01:00
Jan Safranek
989e391d08 Move all volume reconstruction code into separate files
There is no code change, just moving code around and preparing for the
subsequent commit.
2022-11-02 15:58:21 +01:00
Kubernetes Prow Robot
2d14d50b31
Merge pull request #113406 from jsafrane/fix-selinux-check-of-mounted
Fix SELinux check of mounted volumes
2022-11-01 04:14:45 -07:00
Jan Safranek
d37808faae Report error on a pod startup on SELinux mismatch
When a volume is already mounted with an unexpected SELinux label,
kubelet must unmount it first and then mount it back with the expected one.
Report an error to user, just in case the unmount takes too long.

In therory, this error should not happen too often, because two Pods with
different SELinux label will not enter Desired State of World, see
dsw.AddPodToVolume. It can happen when DSW and ASW SELinux labels only when
a volume has been deleted from DSW (= Pod was deleted) or a volume was
reconstructed after kubelet restart. In both cases, volume manager should
unmount the volume quickly.
2022-10-31 13:59:23 +01:00
Jan Safranek
805482413a Fix SELinux check of mounted volumes
In PodExistsInVolume with volumeObj.seLinuxMountContext != nil we know that
the volume has been previously mounted with a given SELinuxMountContext.

Either it has been mounted by this kubelet and we know it's correct or it
was by a previous instance of kubelet and the context has been
reconstructed from the filesystem. In both cases, the actual context is
correct, regardless if the volume plugin or PV access mode supports SELinux
mounts.
2022-10-31 13:39:48 +01:00
Jan Safranek
a910d83070 Reduce log noise on SELinux mount mismatch
The Desired State of World can require a different SELinux mount context than
is in the Actual State of World and it's perfectly OK. For example when
user changes SELinux context of Pods or when the context is reconstructed
after kubelet restart.

Don't spam log and don't report errors to the user as event - reconciler
will do the right thing and unmount the old volume (with wrong context) and
mount a new one in the next reconciliation. It's not an error, it's
expected workflow.
2022-10-27 18:00:42 +02:00
weizhichen
5d514601a8 gofmt 2022-10-13 01:47:08 +00:00
weizhichen
f2e7211ab8 delete stale code in kubelet volumemanager 2022-08-23 23:36:09 +00:00
Niu Lechuan
24614f8551 Add unit test in volumemanager: Detach a volume that had been mounted by pod should be skipped
Signed-off-by: Niu Lechuan <lechuan.niu@daocloud.io>
2022-08-05 09:03:21 +08:00
Jan Safranek
f9c7ce5b9c Add unit tests for DesiredStateOfWorldPopulator 2022-08-04 10:51:59 +02:00
Jan Safranek
260912490e Add a coment about handling same volumes with different contexts 2022-08-04 10:51:56 +02:00
Jan Safranek
a01e720a1a Rename IsRWOP
To be able to update content of the function to other access modes when we
implement SELinux mount for more of them.
2022-08-04 10:51:54 +02:00
Jan Safranek
1490d51028 Remove noisy log
The error would be logged every reconciler sync (100 ms).
2022-08-04 10:51:53 +02:00
Jan Safranek
0793ecee3a Add unit tests for ASW.AddPodToVolume 2022-08-04 10:51:52 +02:00
Jan Safranek
17d850ee0e Add interface for SELinuxOptionsToFileLabel
github.com/opencontainers/selinux/go-selinux needs OS that supports SELinux
and SELinux enabled in it to return useful data, therefore add an interface
in front of it, so we can mock its behavior in unit tests.
2022-08-04 10:51:51 +02:00
Jan Safranek
d9f792633d Add AddPodToVolume unit tests with SELinux 2022-08-04 10:51:50 +02:00
Jan Safranek
8d6b721ddd Extract SELinux context error handling into a common func
Add handlerSELinuxMetricError() which bumps the right metric + either
consumes a SELinux error or lets it propagate up the stack.
2022-08-04 10:51:48 +02:00
Jan Safranek
49148ddfd0 Extract getSELinuxLabel from AddPodToVolume
To keep the function smaller.
2022-08-04 10:51:46 +02:00
Jan Safranek
de7f5b66ed Fix existing unit tests 2022-08-04 10:51:44 +02:00
Jan Safranek
b2e18c0b20 Add metrics for SELinux context mount
Add separate _errors and _warnings to capture volumes that were rejected
from those will be rejected when the feature is expanded to all access
mode.
2022-08-04 10:51:43 +02:00
Jan Safranek
48b0751269 Add SELinux context tracking to volume manager
Both ActualStateOfWorld and DesiredStateOfWorld must track SELinux context
of volume mounts.
2022-08-04 10:51:41 +02:00
Matthew Cary
e5d387c5d6 Upgrade CSIMigrationGCE feature gate to GA
Change-Id: I620bc4913765c0d6562eb1008216a72e8b0a2970
2022-08-02 09:14:27 -07:00
Kubernetes Prow Robot
9ad4c5c0a0
Merge pull request #110670 from gnufied/fix-pod-deletion-terminating
Fix pod stuck in termination state when mount fails or gets skipped after kubelet restart
2022-07-27 06:31:29 -07:00
Davanum Srinivas
a9593d634c
Generate and format files
- Run hack/update-codegen.sh
- Run hack/update-generated-device-plugin.sh
- Run hack/update-generated-protobuf.sh
- Run hack/update-generated-runtime.sh
- Run hack/update-generated-swagger-docs.sh
- Run hack/update-openapi-spec.sh
- Run hack/update-gofmt.sh

Signed-off-by: Davanum Srinivas <davanum@gmail.com>
2022-07-26 13:14:05 -04:00
Hemant Kumar
835e8ccc76 Use CheckAndMarkAsUncertainViaReconstruction for uncertain volumes
Also only remove volumes from skippedDuringReconstruction only if
volume was marked as attached.
2022-07-22 20:11:37 -04:00
Hemant Kumar
6d43345c06 Remove volume from found during reconstruction if mounted
Add unit tests for removing reconstructed volumes from ASOW
2022-07-22 20:04:51 -04:00
Hemant Kumar
b455270f6e Add unit test for verifying if processReconstructedVolumes works as expected 2022-07-22 20:04:51 -04:00
Hemant Kumar
b8257e8c01 Address review comments 2022-07-22 20:04:51 -04:00
Hemant Kumar
eb071c2755 Fix code to process volumes which were skipped during reconstruction 2022-07-22 20:04:51 -04:00
Hemant Kumar
c8b85fb470 Keep track of each pod that uses a volume during reconstruction
Add tests for volume cleaning up
2022-07-22 20:04:51 -04:00