Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
fix TODO:change to a api-server watch
What this PR does / why we need it:
fix TODO (vladimirvivien) instead of polling api-server, change to a api-server watch
Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes#58031
Special notes for your reviewer:
Release note:
NONE
@vladimirvivien,@jsafrane,@saad-ali
I saw some TODO there and I am very interested in fixing it.
Please let me know if it is not inappropriate to do this.
Automatic merge from submit-queue (batch tested with PRs 59441, 58264, 59287, 59396, 59439). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add context to all relevant cloud APIs
**What this PR does / why we need it**:
This adds context to all the relevant cloud provider interface signatures.
Callers of those APIs are currently satisfied using context.TODO().
There will be follow on PRs to push the context through the stack.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#815
**Special notes for your reviewer**:
For an idea of the full scope of this change please look at PR #58532.
**Release note**:
```release-note
Implementers of the cloud provider interface will note the addition of a context to this interface. Trivial code modification will be necessary for a cloud provider to continue to compile.
```
Automatic merge from submit-queue (batch tested with PRs 59441, 58264, 59287, 59396, 59439). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
fix some log param error
this patch fix some log parameter mistakes.
**What this PR does / why we need it**:
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
/release-note-none
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
fix TODO: moving Driver name check in API validation
**What this PR does / why we need it**:
fix TODO in csi_plugin.go ,moving Driver name check in API validation
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#58163
**Special notes for your reviewer**:
**Release note**:
NONE
@vladimirvivien,@jsafrane,@saad-ali
I saw some TODO there and I am very interested in fixing it.
Please let me know if it is not inappropriate to do this.
This adds context to all the relevant cloud provider interface signatures.
Callers of those APIs are currently satisfied using context.TODO().
There will be follow on PRs to push the context through the stack.
For an idea of the full scope of this change please look at PR #58532.
Automatic merge from submit-queue (batch tested with PRs 52942, 58415). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Improve messaging on volume expansion
- we now provide clear message to user what to do when cloudprovider resizing is finished
and file system resizing is needed.
- add a event when resizing is successful
- Use PATCH both in controller-manager and kubelet for updating PVC status
- Remove code duplication between controller-manager and kubelet for updating PVC status
- Only remove conditions that are managed by resize controller
```release-note
Improve messages user gets during and after volume resizing is done.
```
Automatic merge from submit-queue (batch tested with PRs 54680, 59388, 57455). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fix TestPlugin func has two same if code/kind bug
**What this PR does / why we need it**:
Fix TestPlugin func has two same if code
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
This change queries the StorageOS API to retrieve the volume
device dir when a volume is attached, rather than relying on
the default /var/lib/storageos/volumes directory. This allows
all or some nodes to have kubelet running in a container with
the StorageOS volume path mounted in, which is required for
Azure ACS support.
Volume mount options are also supported.
Automatic merge from submit-queue (batch tested with PRs 58317, 58687, 57872, 59063, 59328). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Volume metrics support for vSphere Cloud Provider
**What this PR does / why we need it**:
This PR adds Volume metrics support for vSphere Cloud Provider. With this we can monitor the available space, capacity and used space on volumes created using vSphere.
**Which issue(s) this PR fixes**
Fixes # https://github.com/kubernetes/kubernetes/issues/43527
Fixes # https://github.com/vmware/kubernetes/issues/208
**Special notes for your reviewer**:
Performed following steps to ensure metrics are reported correctly for vSphere Volumes.
- Created Dynamic PV using the PV claim of size 1Gi.
```
$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
thinclaim Bound pvc-bbc18f6a-0a04-11e8-bb1c-0050569c126b 1Gi RWO thin 5m
```
- Created Pod using this claim.
- Verified volume stats metrics.
```
/ # df -h | grep /dev/sdb
Filesystem Size Used Available Use% Mounted on
/dev/sdb 975.9M 1.3M 923.4M 0% /mnt/volume1
```
```
# HELP kubelet_volume_stats_available_bytes Number of available bytes in the volume
# TYPE kubelet_volume_stats_available_bytes gauge
kubelet_volume_stats_available_bytes{namespace="default",persistentvolumeclaim="thinclaim"} 9.68261632e+08
# HELP kubelet_volume_stats_capacity_bytes Capacity in bytes of the volume
# TYPE kubelet_volume_stats_capacity_bytes gauge
kubelet_volume_stats_capacity_bytes{namespace="default",persistentvolumeclaim="thinclaim"} 1.02330368e+09
# HELP kubelet_volume_stats_used_bytes Number of used bytes in the volume
# TYPE kubelet_volume_stats_used_bytes gauge
kubelet_volume_stats_used_bytes{namespace="default",persistentvolumeclaim="thinclaim"} 1.355776e+06
```
- Download 512MB file in the Pod on attached volume and Verified volume stats metrics is updated successfully.
```
/mnt/volume1 # df -h | grep /dev/sdb
Filesystem Size Used Available Use% Mounted on
/dev/sdb 975.9M 513.8M 410.9M 56% /mnt/volume1
```
```
# HELP kubelet_volume_stats_available_bytes Number of available bytes in the volume
# TYPE kubelet_volume_stats_available_bytes gauge
kubelet_volume_stats_available_bytes{namespace="default",persistentvolumeclaim="thinclaim"} 4.30862336e+08
# HELP kubelet_volume_stats_capacity_bytes Capacity in bytes of the volume
# TYPE kubelet_volume_stats_capacity_bytes gauge
kubelet_volume_stats_capacity_bytes{namespace="default",persistentvolumeclaim="thinclaim"} 1.02330368e+09
# HELP kubelet_volume_stats_used_bytes Number of used bytes in the volume
# TYPE kubelet_volume_stats_used_bytes gauge
kubelet_volume_stats_used_bytes{namespace="default",persistentvolumeclaim="thinclaim"} 5.38755072e+08
```
**Release note**:
```release-note
Volume metrics support for vSphere Cloud Provider
```
This PR is the first part of redesign of volume reconstruction work. The
changes include
1. Remove dependency on volume spec stored in actual state for volume
cleanup process (UnmountVolume and UnmountDevice)
Modify AttachedVolume struct to add DeviceMountPath so that volume
unmount operation can use this information instead of constructing from
volume spec
2. Modify reconciler's volume reconstruction process (syncState). Currently workflow
is when kubelet restarts, syncState() is only called once before
reconciler starts its loop.
a. If volume plugin supports reconstruction, it will use the
reconstructed volume spec information to update actual state as before.
b. If volume plugin cannot support reconstruction, it will use the
scanned mount path information to clean up the mounts.
In this PR, all the plugins still support reconstruction (except
glusterfs), so reconstruction of some plugins will still have issues.
The next PR will modify those plugins that cannot support reconstruction
well.
This PR addresses issue #52683, #54108 (This PR includes the changes to
update devicePath after local attach finishes)
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
add PV size grow feature for azure file
**What this PR does / why we need it**:
According to kubernetes/features#284, add size grow feature for azure file
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#56462
**Special notes for your reviewer**:
Since azure file is using SMB 3.0 protocal, there is no necessary to resize filesystem on agent side, the agent node will detect the changed size automatically.
**Release note**:
```
add size grow feature for azure file
```
/sig azure
@gnufied @rootfs @brendandburns
Automatic merge from submit-queue (batch tested with PRs 59097, 57076, 59295). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
delete duplicate function for getting volume source
**What this PR does / why we need it**:
these two methods have same function.
```go
func getVolumeSource(
spec *volume.Spec) (*v1.GlusterfsVolumeSource, bool, error) {
if spec.Volume != nil && spec.Volume.Glusterfs != nil {
return spec.Volume.Glusterfs, spec.Volume.Glusterfs.ReadOnly, nil
} else if spec.PersistentVolume != nil &&
spec.PersistentVolume.Spec.Glusterfs != nil {
return spec.PersistentVolume.Spec.Glusterfs, spec.ReadOnly, nil
}
return nil, false, fmt.Errorf("Spec does not reference a Glusterfs volume type")
}
func (plugin *glusterfsPlugin) getGlusterVolumeSource(spec *volume.Spec) (*v1.GlusterfsVolumeSource, bool) {
// Glusterfs volumes used directly in a pod have a ReadOnly flag set by the pod author.
// Glusterfs volumes used as a PersistentVolume gets the ReadOnly flag indirectly through the persistent-claim volume used to mount the PV
if spec.Volume != nil && spec.Volume.Glusterfs != nil {
return spec.Volume.Glusterfs, spec.Volume.Glusterfs.ReadOnly
}
return spec.PersistentVolume.Spec.Glusterfs, spec.ReadOnly
}
```
`getVolumeSource` seems to be much better.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 55606, 59185, 58763, 59072, 59251). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Cleaning up loopback removal process
Handling more graceful the process of loopback removal. In case loopback device does not exist any longer, we should not fail the detachment process.
```release-note
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Perform resize of mounted volume if necessary
Under certain conditions - we must perform resize of volume even when it is mounted. This enables us to get around problem of resizing volumes used by deployments etc.
```release-note
Allow expanding mounted volumes
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Postpone PV deletion with finalizer when it is being used
Postpone PV deletion if it is bound to a PVC
xref: https://github.com/kubernetes/community/pull/1608
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#33355
**Special notes for your reviewer**:
**Release note**:
```release-note
Postpone PV deletion when it is being bound to a PVC
```
WIP, assign to myself first
/assign @NickrenREN
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add V1beta1 VolumeAttachment API
**What this PR does / why we need it**:
Add V1beta1 VolumeAttachment API, co-existing with Alpha API object
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#58461
**Special notes for your reviewer**:
**Release note**:
```release-note
Add V1beta1 VolumeAttachment API, co-existing with Alpha API object
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
csi: Update version comparison model
**What this PR does / why we need it**:
CSI version matching needed to be updated to be able to support different patch levels during 0.x.x versions, and different minor.patch levels during >=1.x.x versions.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#58813
```release-note
NONE
```
- we now provide clear message to user what to do when cloudprovider resizing is finished
and file system resizing is needed.
- add a event when resizing is successful.
- Use Patch for updating PVCs in both kubelet and controller-manager
- Extract updating pvc util function in one place.
- Only update resize conditions on progress
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
remove duplicated check of device path in aws attacher
**What this PR does / why we need it**:
The `devicePath` parameter is already validated in this [code](b7100f1ee7/pkg/volume/aws_ebs/attacher.go (L158)), so no need to check it again in the `for loop` as it won't be modified.
This can make the code clearer.
**Release note**:
```release-note
NONE
```
/sig storage
/kind cleanup
If our supported version is still at 0.X.X, then check
also the minor number. If our supported version is >= 1.X.X
then check only the major number.
Closes#58813
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Make ExpandVolumeDevice() idempotent if existing volume capacity meets the requested size.
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
**What this PR does / why we need it**:
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add binary configmap
Reviving code from https://github.com/kubernetes/kubernetes/pull/33549 submitted by @zreigz
**What this PR does / why we need it**:
Add support for binary files in ConfigMap
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#32432
**Special notes for your reviewer**:
**Release note**:
```release-note
ConfigMap objects now support binary data via a new `binaryData` field. When using `kubectl create configmap --from-file`, files containing non-UTF8 data will be placed in this new field in order to preserve the non-UTF8 data. Use of this feature requires 1.10+ apiserver and kubelets.
```
Automatic merge from submit-queue (batch tested with PRs 58626, 58791). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Use correct pv annotation to fetch volume ID.
Fix , Issue #58627
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
**What this PR does / why we need it**:
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
```
Automatic merge from submit-queue (batch tested with PRs 58480, 58549). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Use backup location to load cloud config for OpenStack
**What this PR does / why we need it**:
Since we are transitioning to external cloud provider, we need a way
to use the existing cinder volume plugin (from kubelet). With external
cloud manager kubelet will be run with --cloud=provider=external and
no --cloud-config file will be in the command line. So we need a way
to load the openstack config file from somewhere.
Taking a cue from kubeadm, which currently is picking up "/etc/kubernetes/cloud-config"
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/phases/controlplane/manifests.go#L44
let's support the scenario where we fall back to this static location if
there is no cloud provider specified in the command line.
This has been tested with local-up-cluster using the following params:
EXTERNAL_CLOUD_PROVIDER=true
CLOUD_PROVIDER=openstack
CLOUD_CONFIG=/etc/kubernetes/cloud-config
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 58438, 58523, 58513, 57948). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
fix csi ut print
**What this PR does / why we need it**:
fix csi ut print
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 58438, 58523, 58513, 57948). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add Namespace to glusterfs custom volume names.
For better isolation of volumes, users/admins would like to include `namespace` in custom volume names. This patch adds that support.
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
**What this PR does / why we need it**:
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
```
