iSCSI volume plugin persists volume metadata into global mount directory,
before it is mounted. Content of the directory is shadowed by the volume
mount. Therefore kubelet should not write metadata to the directory when a
second pod uses the same volume on the same node.
1. The metadata were already persisted before mounting the volume for the
first pod.
2. The global mount directory has the volume mounted, so any write there
would write to the volume, which is undesirable.
This patch removes pkg/util/mount completely, and replaces it with the
mount package now located at k8s.io/utils/mount. The code found at
k8s.io/utils/mount was moved there from pkg/util/mount, so the code is
identical, just no longer in-tree to k/k.
This patch removes mount.Exec entirely and instead uses the common
utility from k8s.io/utils/exec.
The fake exec implementation found in k8s.io/utils/exec differs a bit
than mount.Exec, with the ability to pre-script expected calls to
Command.CombinedOutput(), so tests that previously relied on a callback
mechanism to produce specific output have been updated to use that
mechanism.
We sometimes face issues with iSCSI PVs and it's hard to guess what's going
on without iscsiadm commands logged. Using level 5 (iscsiadm output can be
long sometimes) and explicitly avoiding logging of CHAP passwords.
In addition, log which path failed to appear after timeout, so the admin
can see which portals are not providing devices.
Kubernetes should retry detaching iSCSI volumes on error. In addition, it
should not report an error when detaching a disk while the disk is already
detached.
At present, iscsi plugin wait for 10seconds for a path to appear for a multipath
device, but at certain scenarios this may not be sufficient for device mapper
to get the path. The default multipath configuration has a configuation
called 'checker_timeout' which specify the timeout to user for path checkers
that issue scsi commands with an explicit timeout, in seconds;
default taken from /sys/block/sd*/device/timeout which is 30s.
This patch lift the timeout value from 10s to 30s.
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
This patch cleans up pkg/util/mount/* and pkg/util/volume/* to always
use filepath.Join instead of path.Join. filepath.Join is preferred
because path.Join can have issues on Windows.
- Move from the old github.com/golang/glog to k8s.io/klog
- klog as explicit InitFlags() so we add them as necessary
- we update the other repositories that we vendor that made a similar
change from glog to klog
* github.com/kubernetes/repo-infra
* k8s.io/gengo/
* k8s.io/kube-openapi/
* github.com/google/cadvisor
- Entirely remove all references to glog
- Fix some tests by explicit InitFlags in their init() methods
Change-Id: I92db545ff36fcec83afe98f550c9e630098b3135
Don't mount single path instead of multipath volumes and always wait until
at least 2 paths are available. Try up to 3 times to get all paths. Try 5
times to get at last 2 paths.
It takes a variable amount of time for the multipath daemon
to create /dev/dm-XX in response to new LUNs being discovered.
The old iscsi_util code only discovered the multipath device
if it was created quickly enough, but in a significant number
of cases, kubelet would grab one of the individual paths and
put a filesystem it on before multipathd could construct a
multipath device.
This change waits for the multipath device to get created for
up to 10 seconds, but only if the PV actually had more than
one portal.
This change ensures that iSCSI block devices are deleted after
unmounting, and implements scanning of individual LUNs rather
than scanning the whole iSCSI bus.
In cases where an iSCSI bus is in use by more than one attachment,
detaching used to leave behind phantom block devices, which could
cause I/O errors, long timeouts, or even corruption in the case
when the underlying LUN number was recycled. This change makes
sure to flush references to the block devices after unmounting.
The original iSCSI code scanned the whole target every time a LUN
was attached. On storage controllers that export multiple LUNs on
the same target IQN, this led to a situation where nodes would
see SCSI disks that they weren't supposed to -- possibly dozens or
hundreds of extra SCSI disks. This caused 3 significant problems:
1) The large number of disks wasted resources on the node and
caused a minor drag on performance.
2) The scanning of all the devices caused a huge number of uevents
from the kernel, causing udev to bog down for multiple minutes in
some cases, triggering timeouts and other transient failures.
3) Because Kubernetes was not tracking all the "extra" LUNs that
got discovered, they would not get cleaned up until the last LUN
on a particular target was detached, causing a logout. This led
to significant complications:
In the time window between when a LUN was unintentially scanned,
and when it was removed due to a logout, if it was deleted on the
backend, a phantom reference remained on the node. In the best
case, the phantom LUN would cause I/O errors and timeouts in the
udev system. In the worst case, the backend could reuse the LUN
number for a new volume, and if that new volume were to be
scheduled to a pod with a phantom reference to the old LUN by the
same number, the initiator could get confused and possibly corrupt
data on that volume.
To avoid these problems, the new implementation only scans for
the specific LUN number it expects to see. It's worth noting that
the default behavior of iscsiadm is to automatically scan the
whole bus on login. That behavior can be disabled by setting
node.session.scan = manual
in iscsid.conf, and for the reasons mentioned above, it is
strongly recommended to set that option. This change still works
regardless of the setting in iscsid.conf, and while automatic
scanning will cause some problems, this change doesn't make the
problems any worse, and can make things better in some cases.
These are all flagged by Go 1.11's
more accurate printf checking in go vet,
which runs as part of go test.
Lubomir I. Ivanov <neolit123@gmail.com>
applied ammend for:
pkg/cloudprovider/provivers/vsphere/nodemanager.go
Google's configure-helper.sh script bind-mounts /var/lib/kubelet somewhere
into /home/kubernetes and thus every mount that Kubernetes does is visible
twice in /proc/mounts.
iSCSI and RBD should not rely on counting on entries in /proc/mounts and
unmount device when Kubernetes thinks it's unusued. Kubernetes tracks
the mounts by itself and most of other volume plugins rely on it safely.
This PR add comments for the background why plugin gets loopback
device and removes loopback device even if operation_generator has
same functionality.
If the default iSCSI node.startup is set to automatic, if there is a node failure,
any pods on that node will get rescheduled to another node. If the failed node is
later brought back up it will then try to log back in to any iSCSI sessions it had
prior to the failure, which may no longer exist or may be now in-use by the other
nodes.
It appears most platforms keep the open-iscsi default of node.startup-automatic.
But in case this system-wide setting has been changed, and just to be explicit, this
sets node.startup values for kubernetes controlled volumes to manual.
Closes issue #21305
This PR makes following changes.
- Simplify volume tearDown path for iSCSI and FC using
util.UnmountPath().
- Log lastErr during iscsi connection
If iscsid fails to connect second portal, currently
the error is ignored silently. The lastErr should be
logged to find the root cause of problem.
- Remove iscsi plugin directory after iscsi connection
is successfully closed.
WaitForAttach failed consistently with this error:
Heuristic determination of mount point failed:stat /var/lib/kubelet/plugins/kubernetes.io/iscsi/iface-default/10.128.0.3:3260-iqn.2003-01.org.linux-iscsi.f21.x8664:sn.4b0aae584f7c-lun-0: no such file or directory
We should ignore "no such file or directory" eror, the directory is created
just few lines below.
This PR adds iSCSI initiatorname parameter to ISCSIVolumeSource
to enable automatic configuration of initiator name per volume.
This would allow for more fine grained configuration, and remove
the need to configure the initiator name on the host by
administrator.
fixes: #47311
If iscsiTransport is not tcp, iSCSI plugin tries to
find devicepath using filepath.Glob but never updates
devicepath with the filepath.Glob result.
This patch fixes the problem.
Fixes#47253
