In the KEP 1710 we promised to have all SELinux metrics with access mode
label, so cluster admin is able to distinguish when RWOP volumes are
failing to mount (-> SELinuxMountReadWriteOncePod feature gate must be
disabled) or volumes with any other access modes are failing (->
SELinuxMount feature gate must be disabled).
Adding the label to kubelet is quite straightforward, there were some
changes needed in the e2e test. Now grabMetrics() collects values of all
SELinux related metrics with all labels. It only skips unrelated volume
plugins. And waitForMetricIncrease gets metric with all labels on input, so
it can check that say RWOP metric increased and RWX one did not.
Use device mountable volume, to make it impossible to share the same global
mount with different SELinux contexts.
And fix pod2Name to actually refer to pod2.
volume_manager_selinux_volume_context_mismatch_warnings_total should be
counted only once per volume + pod. The previous location is evaluated
periodically, so bump the metric only when a new pod is added to volume.
Record volume plugin name when a volume in a Pod needs a different
"mount -o context" value than the actually mounted one.
We expect that NFS, CIFS and CephFS volumes would be able to mount such
volumes just fine with multiple "-o context" values.
We know that the block-volume based ones (ext4, xfs, btrfs, ...) cannot do
that.
Therefore want to distinguish the volume plugin in metrics, anything
block-volume based could break an existing application.
During CSI volume reconstruction it's not possible to tell, if the volume
is attachable or not - CSIDriver instance may not be available, because
kubelet may not have connection to the API server at that time.
Adding uncertain state during reconstruction + adding a correct state when
the API server is available.
DesiredStateOfWorld must remember both
- the effective SELinux label to apply as a mount option (non-empty for
RWOP volumes, empty otherwise)
- and the label that _would_ be used if the mount option would be used by
all access modes.
Mismatch warning metrics must be generated from the second label.
SELinux context discovered from Pod is not final, it can be cleared when a
volume plugin does not support SELinux or the volume is not
ReadWriteOncePod. Update the existing log line + add a new one for easier
debugging.
Subsequent SELinux work (see http://kep.k8s.io/1710) will need
ActualStateOfWorld populated around the time kubelet starts mounting
volumes.
Therefore reconstruct volumes before starting reconciler, but do not depend
on the desired state of world populated nor node.status - both need a
working API server, which may not be available at that time.
All reconstructed volumes are marked as Uncertain and reconciler will sort
them out - call SetUp to ensure the volume is really mounted when a pod
needs the volume or call TearDown then there is no such pod.
Finish the reconstruction when the API server becomes available:
- Clean up volumes that failed reconstruction and are not needed.
- Update devicePath of reconstructed volumes from node.status. Make sure
not to overwrite devicePath that may have been updated when the volume
was mounted by reconcile().
Hiding all this rework behind SELinuxMountReadWriteOncePod FeatureGate,
just to make sure we have a way back if this commit is buggy.
When a volume is already mounted with an unexpected SELinux label,
kubelet must unmount it first and then mount it back with the expected one.
Report an error to user, just in case the unmount takes too long.
In therory, this error should not happen too often, because two Pods with
different SELinux label will not enter Desired State of World, see
dsw.AddPodToVolume. It can happen when DSW and ASW SELinux labels only when
a volume has been deleted from DSW (= Pod was deleted) or a volume was
reconstructed after kubelet restart. In both cases, volume manager should
unmount the volume quickly.
In PodExistsInVolume with volumeObj.seLinuxMountContext != nil we know that
the volume has been previously mounted with a given SELinuxMountContext.
Either it has been mounted by this kubelet and we know it's correct or it
was by a previous instance of kubelet and the context has been
reconstructed from the filesystem. In both cases, the actual context is
correct, regardless if the volume plugin or PV access mode supports SELinux
mounts.
github.com/opencontainers/selinux/go-selinux needs OS that supports SELinux
and SELinux enabled in it to return useful data, therefore add an interface
in front of it, so we can mock its behavior in unit tests.
Update mounter interface in volume manager's ActualStateOfWorld every time.
Otherwise kubelet uses the first mounter it gets, which may not have the
latest information.
This fixes set up of CSI volumes, which store information about SELinux
support in their `mounter` interface implementation. With each MountVolume()
retry, a new mounter is instantiated and only the final mounter that succeeds
has the right info if the volume supports SELinux or not and can later
return the right attributes on GetAttributes() call.
