Automatic merge from submit-queue
Azuredisk mount on windows node
**What this PR does / why we need it**:
This PR will enable azure disk on windows node, customer could create a pod mounted with azure disk on windows node.
There are a few pending items still left:
1) Current fstype would be forced as NTFS, will change if there is such requirement
2) GetDeviceNameFromMount function is not implemented(empty) because in Linux, we could use "cat /proc/mounts" to read all mounting points in OS easily, but in Windows, there is no such place, I am still figuring out. The empty function would cause a few warning logging, but it will not affect the main logic now.
**Special notes for your reviewer**:
1. This PR depends on https://github.com/kubernetes/kubernetes/pull/51240, which allow windows mount path in config validation
2. There is a bug in docker on windows(https://github.com/moby/moby/issues/34729), the ContainerPath could only be a drive letter now(e.g. D:), dir path would fail in the end.
The example pod with mount path is like below:
```
kind: Pod
apiVersion: v1
metadata:
name: pod-uses-shared-hdd-5g
labels:
name: storage
spec:
containers:
- image: microsoft/iis
name: az-c-01
volumeMounts:
- name: blobdisk01
mountPath: 'F:'
nodeSelector:
beta.kubernetes.io/os: windows
volumes:
- name: blobdisk01
persistentVolumeClaim:
claimName: pv-dd-shared-hdd-5
```
**Release note**:
```release-note
Automatic merge from submit-queue (batch tested with PRs 51041, 52297, 52296, 52335, 52338)
Fix pagesize mount option name
**What this PR does / why we need it**:
Fixes#52337 .
add initial work for mount azure file on windows
fix review comments
full implementation for attach azure file on windows node
working azure file mount
remove useless functions
add a workable implementation about mounting azure file on windows node
fix review comments and make the pod creating successful even azure file mount failed
fix according to review comments
add mount_windows_test
add implementation for IsLikelyNotMountPoint func
remove mount_windows_test.go temporaly
add back unit test for mount_windows.go
add normalizeWindowsPath func
fix normalizeWindowsPath func issue
implment azure disk on windows
update bazel BUILD
revert validation.go change as it's another PR
fix merge issue and compiling issue
fix windows compiling issue
fix according to review comments
fix according to review comments
fix cross-build failure
fix according to review comments
fix test build failure temporalily
fix darwin build failure
fix azure windows test failure
add empty implementation of MakeRShared on windows
fix gofmt errors
As per glusterfs documentation it can't create
volumes in GiB and all sizes must be specified in GB.
This code was slightly buggy because we were creating
volumes of sizes lesser than user asked for
The commit allow ScaleIO volume plugin to read SDC GUID value as a node label.
If binary drv_cfg is not installed, the plugin will still work properly.
If node label not found, it defaults to drv_cfg if installed.
Modifies the VolumeZonePredicate to handle a PV that belongs to more
then one zone or region. This is indicated by the zone or region label
value containing a comma separated list.
Automatic merge from submit-queue (batch tested with PRs 51819, 51706, 51761, 51818, 51500)
Fix local storage code to follow go style
**What this PR does / why we need it**:
Fix local storage code to follow go style
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 51805, 51725, 50925, 51474, 51638)
Flexvolume dynamic plugin discovery: Prober unit tests and basic e2e test.
**What this PR does / why we need it**: Tests for changes introduced in PR #50031 .
As part of the prober unit test, I mocked filesystem, filesystem watch, and Flexvolume plugin initialization.
Moved the filesystem event goroutine to watcher implementation.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#51147
**Special notes for your reviewer**:
First commit contains added functionality of the mock filesystem.
Second commit is the refactor for moving mock filesystem into a common util directory.
Third commit is the unit and e2e tests.
**Release note**:
```release-note
NONE
```
/release-note-none
/sig storage
/assign @saad-ali @liggitt
/cc @mtaufen @chakri-nelluri @wongma7
Automatic merge from submit-queue (batch tested with PRs 51628, 51637, 51490, 51279, 51302)
Fix iSCSI WaitForAttach not mounting a volume
WaitForAttach failed consistently with this error:
```
Heuristic determination of mount point failed:stat /var/lib/kubelet/plugins/kubernetes.io/iscsi/iface-default/10.128.0.3:3260-iqn.2003-01.org.linux-iscsi.f21.x8664:sn.4b0aae584f7c-lun-0: no such file or directory
```
We should ignore "no such file or directory" eror, the directory is created just few lines below.
Fixes: #51625
**Release note**:
```release-note
NONE
```
/sig storage
/assign @mtanino
Automatic merge from submit-queue (batch tested with PRs 51574, 51534, 49257, 44680, 48836)
rbd: default image format to v2 instead of deprecated v1
**What this PR does / why we need it**:
Image format v1 has been deprecated since the Infernalis release of
Ceph over two years ago.
**Release note**:
```StorageClass Ceph RBD now defaults to using the v2 image format ```
WaitForAttach failed consistently with this error:
Heuristic determination of mount point failed:stat /var/lib/kubelet/plugins/kubernetes.io/iscsi/iface-default/10.128.0.3:3260-iqn.2003-01.org.linux-iscsi.f21.x8664:sn.4b0aae584f7c-lun-0: no such file or directory
We should ignore "no such file or directory" eror, the directory is created
just few lines below.
Automatic merge from submit-queue (batch tested with PRs 51439, 51361, 51140, 51539, 51585)
Iscsi handle failed mount
**What this PR does / why we need it**:
**Which issue this PR fixes**: fixes#50556
**Special notes for your reviewer**:
**Release note**:
```release-note
None
```
Automatic merge from submit-queue (batch tested with PRs 50919, 51410, 50099, 51300, 50296)
Take mount options to GA by adding PV.spec.mountOptions
**What this PR does / why we need it**: Implements https://github.com/kubernetes/community/pull/771
issue: https://github.com/kubernetes/features/issues/168
**Special notes for your reviewer**:
TODO:
- ~StorageClass mountOptions~
As described in proposal, this adds PV.spec.mountOptions + mountOptions parameter to every plugin that is both provisionable & supports mount options.
(personally, even having done all the work already, i don't agree w/ the proposal that mountOptions should be SC parameter but... :))
**Release note**:
```release-note
Add mount options field to PersistentVolume spec
```
Automatic merge from submit-queue
Add volume operation metrics to operation executor and PV controller
This PR implements the proposal for high level volume metrics https://github.com/kubernetes/community/pull/809
**Special notes for your reviewer**:
~Differences from proposal:~ all resolved
~"verify_volume" is now "verify_volumes_are_attached" + "verify_volumes_are_attached_per_node" + "verify_controller_attached_volume." Which of them do we want?~
~There is no "mount_device" metric because the MountVolume operation combines MountDevice and mount (plugin.Setup). Do we want to extract the mount_device metric or is it okay to keep mountvolume as one? For attachable volumes, MountDevice is the actual mount and Setup is a bindmount + setvolumeownership. For unattachable, mountDevice does not occur and Setup is an actual mount + setvolumeownership.~
~PV controller metrics I did not implement following the proposal at all. I did not change goroutinemap nor scheduleOperation. Because provisionClaimOperation does not return an error, so it's impossible for the caller to know if there is actually a failure worth reporting. So I manually create a new metric inside the function according to some conditions.~
@gnufied
I have tested the operationexecutor metrics but not provision & delete. Sample:
**Release note**:
```release-note
Add error count and time-taken metrics for storage operations such as mount and attach, per-volume-plugin.
```
Automatic merge from submit-queue (batch tested with PRs 49849, 50334, 51414)
Support iscsi volume attach and detach
**What this PR does / why we need it**:
Support iSCSI volume attach and detach to enforce RWO access
**Which issue this PR fixes** : fixes#48953
**Special notes for your reviewer**:
**Release note**:
```
NONE
```
This change is prerequisite for implementing iSCSI attacher
and detacher.
In order to use chap authentication at iSCSI plugin after
implementing attacher and detacher, secret is needed at
AttachDisk() which is called from WaitForAttach().
To obtain secret, pod information is required, but
WaitForAttach() doesn't pass pod information inside.
This patch adds 'pod' as an argument of WaitForAttach()
and adds changes to drivers who implements WaitForAttach().
Fixes#48953
Automatic merge from submit-queue
Fix swallowed error in iscsi package
**What this PR does / why we need it**: Fixes a swallowed error in the iscsi package.
```release-note NONE
```
Automatic merge from submit-queue
Fix swallowed errors in tests of photon_pd package
**What this PR does / why we need it**: Fixes swallowed errors in the tests of the photon_pd package.
```release-note NONE
```
Automatic merge from submit-queue (batch tested with PRs 51174, 51363, 51087, 51382, 51388)
Fix swallowed error in scaleio package tests
**What this PR does / why we need it**: Fixes a dropped error in the tests of the scaleio package.
**Release note**:
```release-note NONE
```
Automatic merge from submit-queue (batch tested with PRs 51174, 51363, 51087, 51382, 51388)
Add InstanceExistsByProviderID to cloud provider interface for CCM
**What this PR does / why we need it**:
Currently, [`MonitorNode()`](02b520f0a4/pkg/controller/cloud/nodecontroller.go (L240)) in the node controller checks with the CCM if a node still exists by calling `ExternalID(nodeName)`. `ExternalID` is supposed to return the provider id of a node which is not supported on every cloud. This means that any clouds who cannot infer the provider id by the node name from a remote location will never remove nodes that no longer exist.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#50985
**Special notes for your reviewer**:
We'll want to create a subsequent issue to track the implementation of these two new methods in the cloud providers.
**Release note**:
```release-note
Adds `InstanceExists` and `InstanceExistsByProviderID` to cloud provider interface for the cloud controller manager
```
/cc @wlan0 @thockin @andrewsykim @luxas @jhorwit2
/area cloudprovider
/sig cluster-lifecycle
Automatic merge from submit-queue (batch tested with PRs 51054, 51101, 50031, 51296, 51173)
Dynamic Flexvolume plugin discovery, probing with filesystem watch.
**What this PR does / why we need it**: Enables dynamic Flexvolume plugin discovery. This model uses a filesystem watch (fsnotify library), which notifies the system that a probe is necessary only if something changes in the Flexvolume plugin directory.
This PR uses the dependency injection model in https://github.com/kubernetes/kubernetes/pull/49668.
**Release Note**:
```release-note
Dynamic Flexvolume plugin discovery. Flexvolume plugins can now be discovered on the fly rather than only at system initialization time.
```
/sig-storage
/assign @jsafrane @saad-ali
/cc @bassam @chakri-nelluri @kokhang @liggitt @thockin
Automatic merge from submit-queue (batch tested with PRs 51235, 50819, 51274, 50972, 50504)
handle failed mounts for fc volumes
**What this PR does / why we need it**: handles failed mounts for fc
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#50502
**Special notes for your reviewer**:
**Release note**:
```release-note
None
```
Automatic merge from submit-queue (batch tested with PRs 51235, 50819, 51274, 50972, 50504)
Clean cinder detachlogerror
**What this PR does / why we need it**:
**Which issue this PR fixes** : fixes#50441
**Special notes for your reviewer**:
**Release note**:
```release-note
```
Automatic merge from submit-queue (batch tested with PRs 50033, 49988, 51132, 49674, 51207)
Adding fsGroup check before mounting a volume
fsGroup check will be enforcing that if a volume has already been
mounted by one pod and another pod wants to mount it but has a different
fsGroup value, this mount operation will not be allowed.
Closes#45053
Automatic merge from submit-queue (batch tested with PRs 50872, 51103, 51220, 51285, 50841)
ScaleIO: Use VolumeHost.GetExec() to execute stuff in volume plugins
This PR updates ScaleIO volume plugin to use `VolumeHost.GetExec()` to execute utilities like mkfs and lsblk instead of simple `os/exec` + to use a fresh `mounter` for every `SetUp` / `TearDown` calls, as they may be different each time.
This prepares the volume plugin to run these utilities in containers instead of running them on the host + makes the volume plugin more independent and less hardcoded.
See proposal in https://github.com/kubernetes/community/pull/589.
Note that this PR does **not** change place where the utilities are executed - `VolumeHost.GetExec()` still leads directly to `os/exec`. It will be changed when the aforementioned proposal is merged and implemented.
**Special notes for your reviewer**:
* I needed to pass `mount.Exec` interface from the place where it is available down to `SioClient` where it's needed to execute stuff.
@kubernetes/sig-storage-pr-reviews
/assign @vladimirvivien @rootfs
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 51114, 51233, 51024, 51053, 51197)
rbd: Use VolumeHost.GetExec() to execute stuff in volume plugins
**What this PR does / why we need it**:
This PR updates rbd volume plugin to use `VolumeHost.GetExec()` to execute utilities like mkfs and lsblk instead of simple `os/exec`. This prepares the volume plugin to run these utilities in containers instead of running them on the host + makes the volume plugin more independent and less hardcoded.
See proposal in https://github.com/kubernetes/community/pull/589.
Note that this PR does **not** change place where the utilities are executed - `VolumeHost.GetExec()` still leads directly to `os/exec`. It will be changed when the aforementioned proposal is merged and implemented.
@kubernetes/sig-storage-pr-reviews
**Release note**:
```release-note
NONE
```
