Automatic merge from submit-queue (batch tested with PRs 53694, 53919). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
fix controller manager crash issue on a manually created k8s cluster
**What this PR does / why we need it**:
fix controller manager crash issue on a manually created k8s cluster, it's due to availability set nil issue in azure loadbalancer
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
In the testing of a manually created k8s cluster, I found controller manager on master would crash in current scenario:
1. Use acs-engine to set up k8s 1.7.7 cluster (it's with an availability set)
2. Manually add a node to the k8s cluster (without an availibity set in this VM)
3. Set up a service and schedule the pod onto this newly added node
4. controller manager would crash on master because although this k8s cluster has an availability set, the newly added node's `machine.AvailabilitySet` is nil which would cause controller manager crash
**Special notes for your reviewer**:
@brendanburns @karataliu @JiangtianLi
**Release note**:
```
fix controller manager crash issue on a manually created k8s cluster
```
/sig azure
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Support autoprobing node-security-group for openstack cloud provider
1. Support autoprobing node-security-group
2. Support multiple Security Groups for cluster's nodes
3. Fix recreating Security Group for cluster's nodes
This is a part of #50726
**Special notes for your reviewer**:
/assign @anguslees
/assign @dims
**Release note**:
```release-note
Support autoprobing node-security-group for openstack cloud provider, Support multiple Security Groups for cluster's nodes.
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fix format specifiers in Azure cloud provider
**What this PR does / why we need it**: Fixes invalid/mismatched format specifiers in Azure cloud provider logging statements (`glog...Infof(...)`) that would cause information to be lost in logging output, as flagged by `go vet`.
**Which issue this PR fixes**: None
**Special notes for your reviewer**: None
**Release note**:
```release-note
NONE
```
Currently the service's name is not unique, and the Securty Group
name is not unique too. openstack cloud provider will delete the
Securty Group of other loadbalancer service when do a deletion.
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
fix minor typo
**What this PR does / why we need it**:
Typo error
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
```
Automatic merge from submit-queue (batch tested with PRs 53444, 52067, 53571, 53182). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Retry when checking Azure storage account readiness
**What this PR does / why we need it**: When the Azure cloud provider ensures that a default storage container exists, if the storage account exists but is still provisioning, it exits without retrying. This is a bug as the code is wrapped in a backoff policy but never signals the policy to retry. This PR fixes this behaviour by returning values which allow the backoff policy to operate.
**Which issue this PR fixes**: fixes#53052
**Special notes for your reviewer**: Not sure how to test this - I have done a deployment using acs-engine and it seems to work but I am not sure of the best way to exercise the failure path.
**Release note**:
```release-note
NONE
```
1. Support autoprobing node-security-group
2. Support multiple Security Groups for cluster's nodes
3. Fix recreating Security Group for cluster's nodes
This is a part of #50726
Automatic merge from submit-queue (batch tested with PRs 52662, 53547, 53588, 53573, 53599). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Return err when delete volume failed
Return err when delete volume failed
**Release note**:
```release-note
NONE
```
/kind bug
/sig openstack
Automatic merge from submit-queue (batch tested with PRs 53567, 53197, 52944, 49593). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
[OpenStack]Add codes to check the count of nodes(members)
After merging this PR(#53146), if there is no available nodes for
the loadbalancer service, UpdateLoadBalancer() will run panic.
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
add possibility to ignore volume label in dynamic provisioning
**What this PR does / why we need it**: this is needed if openstack cinder zone name does not match to compute zone names. For instance if there is only one cinder zone and many compute zones.
**Which issue this PR fixes**: fixes#53488
**Special notes for your reviewer**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 52768, 51898, 53510, 53097, 53058). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Ability to run the openstack tests against DevStack
**What this PR does / why we need it**:
Some of the environment variables have changed as devstack defaults
have changed. So look for the older env variables first and try the
newer ones later.
At a minimum you need the following for v3 authentication which is
the default with latest devstack. If you miss the Tenant information
then the token issued will be a unscoped token (and will not have any
service catalog information).
OS_AUTH_URL=http://192.168.0.42/identity
OS_REGION_NAME=RegionOne
OS_USERNAME=demo
OS_PASSWORD=supersecret
OS_TENANT_NAME=demo
OS_USER_DOMAIN_ID=default
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 53418, 53366, 53115, 53402, 53130). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fix the version detection of OpenStack Cinder
**What this PR does / why we need it**:
When running Kubernetes against an installation of DevStack which
deploys the Cinder service at a path rather than a port (ex:
http://foo.bar/volume rather than http://foo.bar:xxx), the version
detection fails. It is better to use the OpenStack service catalog.
OTOH, when initialize cinder client, kubernetes will check the
endpoint from the OpenStack service catalog, so we can do this
version detection by it.
There are two case should be fixed in other PR:
1. revisit the version detection after supporting Cinder V3 API.
2. add codes to support MicroVersion after gophercloud supports MicroVersion.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#50461
**Special notes for your reviewer**:
/assign @dims
/assign @xsgordon
**Release note**:
```release-note
Using OpenStack service catalog to do version detection
```
Some of the environment variables have changed as devstack defaults
have changed. So look for the older env variables first and try the
newer ones later.
At a minimum you need the following for v3 authentication which is
the default with latest devstack. If you miss the Tenant information
then the token issued will be a unscoped token (and will not have any
service catalog information).
OS_AUTH_URL=http://192.168.0.42/identity
OS_REGION_NAME=RegionOne
OS_USERNAME=demo
OS_PASSWORD=supersecret
OS_TENANT_NAME=demo
OS_USER_DOMAIN_ID=default
Automatic merge from submit-queue (batch tested with PRs 51750, 53195, 53384, 53410). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
GCE: Handle missing subnet for legacy networks and auto networks with unique subnet names
Fixes#53409
/assign @bowei
Tested on three GKE clusters with automatic, manual, and legacy networks.
**Release note**:
```release-note
GCE: Fixes ILB sync on legacy networks and auto networks with unique subnet names
```
Automatic merge from submit-queue (batch tested with PRs 51750, 53195, 53384, 53410). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
add http request timeout for OpenStack cloud provider
**What this PR does / why we need it**:
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#53191
**Special notes for your reviewer**:
/assign @NickrenREN @dims @FengyunPan
**Release note**:
```release-note
None
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Move AWS Fake implementations out of test
The AWS fake implementations are in a test file and can't be imported into any other tests. This makes integration testing difficult. This PR moves the fake implementations such that they can be used by other entities.
@kubernetes/sig-aws-misc @justinsb
Automatic merge from submit-queue (batch tested with PRs 53234, 53252, 53267, 53276, 53107). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
add get alpha backend service into cloud provider
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 53101, 53158, 52165). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
[OpenStack] Service LoadBalancer defaults to external
**What this PR does / why we need it**:
Let "service.beta.kubernetes.io/openstack-internal-load-balancer" default to false.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*:
fixes#53078
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 51311, 52575, 53169). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Unable to detach the vSphere volume from Powered off node
With the existing implementation when a vSphere node is powered off, the node is not deleted by the node controller and is in "NotReady" state. Following the approach similar to GCE as mentioned here - https://github.com/kubernetes/kubernetes/issues/46442.
I observe the following issues:
- The pods on the powered off node are not **instantaneously** created on the other available node. Only after 5 minutes timeout, the pods will be created on other available nodes with the volume attached to it. This means an application downtime of around 5 minutes which is not good at all.
- The volume on the powered off node are not detached at all when the pod with the volume is already moved to other available node. Hence any attempt to restart the powered off node will fail as the same volume is attached to other node which is present on this powered off node. (Please note that the volumes are not automatically detached from powered off in vSphere as opposed to GCE, AWS where volume is automatically detached from when node is powered off).
So inorder to resolve this problem, we have decided to back with the approach where the powered off node will be removed by the Node controller. So the above 2 problems will be resolved as follows:
- Since the node is deleted, the pod on the powered off node becomes instantaneously available on other available nodes with the volume attached to the new nodes. Hence there is no application downtime at all.
- After a period of 6 minutes (timeout period), the volumes are automatically detached from the powered off node. Hence any restarts after 6 minutes on the powered off node would work and not cause any problems as volumes are already detached.
For now, we would want to go ahead with deleting the node from node controller when a node is powered off in vCenter until we have a better approach. I think the best possible solution would be to introduce power handler in volume controller to see if the node is powered off before we can take any appropriate for attach/detach operations.
```release-note
None
```
@jingxu97 @saad-ali @divyenpatel @luomiao @rohitjogvmw
Automatic merge from submit-queue (batch tested with PRs 50280, 52529, 53093, 53108, 53168). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Mark volume as detached when node does not exist for photon
If node does not exist, node's volumes will be detached
automatically and become available. So mark them detached and
return false without error.
Fix#50266
**Special notes for your reviewer**:
/assign @jingxu97
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 53157, 52628). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Added openstack instance metadata search order
**What this PR does / why we need it**: This PR adds a search order for the instance metadata retrieval on openstack. More information and discussion can be found on #52378
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#52378
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
When running Kubernetes against an installation of DevStack which
deploys the Cinder service at a path rather than a port (ex:
http://foo.bar/volume rather than http://foo.bar:xxx), the version
detection fails. It is better to use the OpenStack service catalog.
OTOH, when initialize cinder client, kubernetes will check the
endpoint from the OpenStack service catalog, so we can do this
version detection by it.
Automatic merge from submit-queue (batch tested with PRs 52751, 52898, 52633, 52611, 52609). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>..
Only register floatingIP for external loadbalancer service
If the user has provided the floating-ip options, then it's safe
to assume they want (only) the floating-ip to be the ingress IP;
if they have not provided floating-ip options, then the LB IP is
the only relevant value.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
Fix#52566
**Release note**:
```release-note
Only register floatingIP into Loadbalancer ingress field for external loadbalancer service
```
Automatic merge from submit-queue (batch tested with PRs 52751, 52898, 52633, 52611, 52609). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>..
Fix missing floatingip when calling GetLoadBalancer()
If user specify floating-network-id, a floatingip and a vip will
be assigned to LoadBalancer service, So its status contains a
floatingip and a vip, but GetLoadBalancer() only return vip.
**Release note**:
```release-note
GetLoadBalancer() only return floatingip when user specify floating-network-id, or return LB vip.
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>..
Fix GCE LB resource cleanup for service e2e tests.
**What this PR does / why we need it**: Fix GCE LB resource cleanup logic.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#52347
**Special notes for your reviewer**:
/assign @shyamjvs @nicksardo
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 52880, 52855, 52761, 52885, 52929). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>..
Remove cloud provider rackspace
**What this PR does / why we need it**:
For now, we have to implement functions in both `rackspace` and `openstack` packages if we want to add function for cinder, for example [resize for cinder](https://github.com/kubernetes/kubernetes/pull/51498). Since openstack has implemented all the functions rackspace has, and rackspace is considered deprecated for a long time, [rackspace deprecated](https://github.com/rackspace/gophercloud/issues/592) ,
after talking with @mikedanese and @jamiehannaford offline , i sent this PR to remove `rackspace` in favor of `openstack`
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#52854
**Special notes for your reviewer**:
**Release note**:
```release-note
The Rackspace cloud provider has been removed after a long deprecation period. It was deprecated because it duplicates a lot of the OpenStack logic and can no longer be maintained. Please use the OpenStack cloud provider instead.
```
Automatic merge from submit-queue (batch tested with PRs 50068, 52406, 52394, 48551, 52131). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>..
Implement bulk polling of volumes for vSphere
This PR implements bulk polling of volumes - BulkVerifyVolumes() API for vSphere.
With the existing implementation, vSphere makes multiple calls to VC to check if the volume is attached to a node. If there are "N" volumes attached on "M" nodes, vSphere makes "N" VCenter calls to check if the volumes are attached to VC for all "N" volumes. Also, by default Kubernetes queries if the volumes are attached to nodes every 1 minute. This will substantially increase the number of calls made by vSphere cloud provider to vCenter.
Inorder to prevent this, vSphere cloud provider implements the BulkVerifyVolumes() API in which only a single call is made to vCenter to check if all the volumes are attached to the respective nodes. Irrespective of the number of volumes attached to nodes, the number of vCenter calls will always be 1 on a query to BulkVerifyVolumes() API by kubernetes.
@rohitjogvmw @divyenpatel @luomiao
```release-note
BulkVerifyVolumes() implementation for vSphere
```
Automatic merge from submit-queue (batch tested with PRs 52355, 52537, 52551, 52403, 50673). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>..
Azure - Use cloud environment to instantiate storage client
**What this PR does / why we need it**:
Since 1.7 and managed disk for azure, blob storage on Azure cloud other than the default public one is broken, because kubernetes expect blob ressources URI to end with `.blob.core.windows.net ` (ignoring storageEndpointSuffix).
This include the chinese Cloud, for which storageEndpointSuffix is `blob.core.chinacloudapi.cn` for example.
See : https://github.com/Azure/azure-storage-go/blob/master/client.go#L194
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 50294, 50422, 51757, 52379, 52014). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>..
Azure cloud provider: expose services on non-default subnets
**What this PR does / why we need it**: The Azure cloud provider allows users to specify that a service should be exposed on an internal load balancer instead of the default external load balancer. However, in a VNet environment, such services are currently always exposed on the master subnet. Where there are multiple subnets in the VNet, it's desirable to be able to expose an internal service on any subnet. This PR allows this via a new annotation, `service.beta.kubernetes.io/azure-load-balancer-internal-subnet`.
**Which issue this PR fixes**: fixes https://github.com/Azure/acs-engine/issues/1296 (no corresponding issue has been raised in the k8s core repo)
**Special notes for your reviewer**: None
**Release note**:
```release-note
A new service annotation has been added for services of type LoadBalancer on Azure,
to specify the subnet on which the service's front end IP should be provisioned. The
annotation is service.beta.kubernetes.io/azure-load-balancer-internal-subnet and its
value is the subnet name (not the subnet ARM ID). If omitted, the default is the
master subnet. It is ignored if the service is not on Azure, if the type is not
LoadBalancer, or if the load balancer is not internal.
```
