Commit Graph

422 Commits

Author SHA1 Message Date
Kubernetes Submit Queue
96ec318718
Merge pull request #59842 from ixdy/update-rules_go-02-2018
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

 Update bazelbuild/rules_go, kubernetes/repo-infra, and gazelle dependencies

**What this PR does / why we need it**: updates our bazelbuild/rules_go dependency in order to bump everything to go1.9.4. I'm separating this effort into two separate PRs, since updating rules_go requires a large cleanup, removing an attribute from most build rules.

**Release note**:

```release-note
NONE
```
2018-02-19 22:23:05 -08:00
Kubernetes Submit Queue
6d0b71740f
Merge pull request #59968 from kubernetes/revert-59323-nodetaint
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Revert "add node shutdown taint"

Reverts kubernetes/kubernetes#59323

Node becomes unready, but is never removed. I've found the following in [kube-controller-manager.log](https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-autoscaling/6055/artifacts/bootstrap-e2e-master/cluster-autoscaler.log) from test run for one such node:

`E0216 01:14:27.084923       1 node_lifecycle_controller.go:686] Error determining if node bootstrap-e2e-minion-group-01b1 shutdown in cloud: failed to get instance ID from cloud provider: instance not found`

This goes on for the rest of the run (~6h). Looks like the node is stuck in Unready state because of this check: https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/nodelifecycle/node_lifecycle_controller.go#L684. Previously, there was no such check and the node was removed.

Reverting as this would affect all users attempting to resize their node groups on GCE.

```release-note
NONE
```
2018-02-16 20:12:56 -08:00
Jeff Grafton
ef56a8d6bb Autogenerated: hack/update-bazel.sh 2018-02-16 13:43:01 -08:00
Kubernetes Submit Queue
0e81651e77
Merge pull request #59909 from jsafrane/volumemanager-approvers
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Add jsafrane as AWS approver.

**What this PR does / why we need it**:
I contrinbuted several PRs in AWS storage and I'm willing to share review/approval duty.

**Release note**:

```release-note
NONE
```

/assign @justinsb
2018-02-16 06:02:01 -08:00
Aleksandra Malinowska
2d54ba3e0f
Revert "add node shutdown taint" 2018-02-16 12:24:27 +01:00
Bryce Carman
3b99e1b487 Add AWS cloud provider option for IAM role
Currently the AWS cloud provider uses the EC2 instance role when
interacting with AWS APIs. This change gives the option to provide and IAM
role that the cloud provider will assume before calling the APIs. All
resources created by the role will be owned by that account instead of
the account where the EC2 instance is running.
2018-02-15 21:14:58 -08:00
Kubernetes Submit Queue
27daaab224
Merge pull request #59323 from zetaab/nodetaint
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

add node shutdown taint

**What this PR does / why we need it**: we need node stopped taint in order to detach volumes immediately without waiting timeout. More info in issue ticket #58635 

**Which issue(s) this PR fixes** 
Fixes #58635

**Special notes for your reviewer**:

**Release note**:
```release-note
NONE
```
2018-02-15 09:52:10 -08:00
Jan Safranek
161972272c Add jsafrane as AWS approver. 2018-02-14 17:41:18 +01:00
Kubernetes Submit Queue
757c24d224
Merge pull request #57969 from jsafrane/aws-approver
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Add gnufied as AWS approver.

@gnufied has been maintaining the storage part of AWS cloud provider for a long while and he deserves to be approver.

```release-note
NONE
```

/sig aws
2018-02-12 19:41:02 -08:00
Kubernetes Submit Queue
30f54b4028
Merge pull request #56974 from gnufied/speed-up-attach-detach
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Make AWS attach/detach operations faster

Most attach/detach operations on AWS finish within 1-4seconds.
By using a shorter time interval and higher exponetial
factor  we can shorten time taken for attach and detach to complete. 

After this change retry interval looks like:

```
[1, 1.8, 3.24, 5.832000000000001, 10.4976]
```

Before it was:
```
[10, 12.0, 14.399999999999999, 17.279999999999998]
```


/sig aws

```release-note
AWS: Make attach/detach operations faster. from 10-12s to 2-6s
```
2018-02-08 20:24:10 -08:00
Tomas Smetana
c879e531f9 AWS: Do not ignore errors from EC2::DescribeVolumee in DetachDisk
The DetachDisk method of AWS cloudprovider indirectly calls
EC2::DescribeVolume AWS API function to check if the volume
being detached is really attached to the specified node.

The AWS API call may fail and return error which is logged however
the DetachDisk then finishes successfully. This may cause the AWS
volumes to remain attached to the instances forever because the
attach/detach controller will mark the volume as attached. The PV
controller will never be able to delete those disks and they need
to be detached manually.

This patch ensures on error from DescribeVolume is propagated to
attach/detach controller and the detach operation is re-tried.
2018-02-08 13:51:53 +01:00
Jesse Haka
3cf5b172fa add node shutdown taint
shutdowned -> stopped

use shutdown everywhere

use patch in taints api call

use notimplemented in clouds use AddOrUpdateTaintOnNode

correct log text

add fake cloud

try to fix bazel

add shutdown tests

add context
2018-02-08 12:56:06 +02:00
Walter Fender
e18e8ec3c0 Add context to all relevant cloud APIs
This adds context to all the relevant cloud provider interface signatures.
Callers of those APIs are currently satisfied using context.TODO().
There will be follow on PRs to push the context through the stack.
For an idea of the full scope of this change please look at PR #58532.
2018-02-06 12:49:17 -08:00
Kubernetes Submit Queue
eff07b6f55
Merge pull request #56759 from aledbf/fix-nlb-icmp
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Fix NLB icmp permission duplication

**What this PR does / why we need it**:

Fixes an issue with the ICMP rule for MTU during the creation of a NLB

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:

Fixes #56703
2018-01-09 12:20:29 -08:00
Jan Safranek
1a2e651a1c Add gnufied as AWS approver. 2018-01-08 14:21:24 +01:00
Kubernetes Submit Queue
21a15c5673
Merge pull request #50923 from rowleyaj/typo_fix
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Fix typo in comment
2018-01-07 09:03:45 -08:00
Andrew Pilloud
a57d492713 Move DefaultMaxEBSVolumes constant into scheduler
A constant only used by the scheduler lives in the aws cloudprovider
package. Moving the constant into the only package where it is used
reduces import bloat.
2018-01-03 08:19:38 -08:00
Jeff Grafton
efee0704c6 Autogenerate BUILD files 2017-12-23 13:12:11 -08:00
Kubernetes Submit Queue
2ae99cf146
Merge pull request #56955 from feiskyer/scrub-dns
Automatic merge from submit-queue (batch tested with PRs 56997, 57008, 56984, 56975, 56955). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Remove unused ScrubDNS interface from cloudprovider

**What this PR does / why we need it**:

DNS scrubber from kubelet has been removed in #36785 and cloudprovider's `ScrubDNS()` interface is not used anywhere.

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #56953.

**Special notes for your reviewer**:

**Release note**:

```release-note
Remove ScrubDNS interface from cloudprovider.
```
2017-12-16 15:23:54 -08:00
chrislovecnm
4a16f16af4 Adding myself as a reviewer to aws cloud provider 2017-12-13 22:42:07 -07:00
Manuel de Brito Fontes
bea2d8d1aa Fix NLB icmp permission duplication 2017-12-13 20:31:09 -03:00
Hemant Kumar
0e6a541036 Make AWS attach/detach operations faster
Most attach/detach operations on AWS finish within 1-4seconds.
By using a shorter time interval and higher exponetial
factor  we can shorten time taken for attach and detach to complete.
2017-12-08 15:28:58 -05:00
Pengfei Ni
65efeee64f Remove unused ScrubDNS interface from cloudprovider 2017-12-08 16:03:56 +08:00
Hemant Kumar
514f219c22 cloud-provider needs cluster-role to apply taint to the node
When volume is stuck in attaching state on AWS, cloud-provider
needs to taint the node. But the node can not be tainted
without proper access.
2017-12-04 10:57:21 -05:00
Justin Santa Barbara
8bfb676378 AWS: Support for mounting nvme volumes 2017-11-30 14:48:33 -05:00
Hemant Kumar
ac2c68ad8f AWS: Implement fix for detaching volume from stopped instances
Clean up detach disk functions and remove duplication
2017-11-23 11:02:09 -05:00
Hemant Kumar
8c49d1db02 Implement disk resizing for AWS
Update bazel files
2017-11-22 21:38:54 -05:00
Kubernetes Submit Queue
6b1b6d504a
Merge pull request #56024 from dimpavloff/aws-elb-set-hc-params
Automatic merge from submit-queue (batch tested with PRs 56211, 56024). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

allow ELB Healthcheck configuration via Service annotations

**What this PR does / why we need it**:
The default settings which are set on the ELB HC work well but there are cases when it would be better to tweak its parameters -- for example, faster detection of unhealthy backends. This PR makes it possible to override any of the healthcheck's parameters via annotations on the Service, with the exception of the Target setting which continues to be inferred from the Service's spec.

**Release note**:
```release-note
It is now possible to override the healthcheck parameters for AWS ELBs via annotations on the corresponding service. The new annotations are `healthy-threshold`, `unhealthy-threshold`, `timeout`, `interval` (all prefixed with `service.beta.kubernetes.io/aws-load-balancer-healthcheck-`)
```
2017-11-22 08:48:43 -08:00
Kubernetes Submit Queue
63d4b85bf4
Merge pull request #53400 from micahhausler/aws-nlb
Automatic merge from submit-queue (batch tested with PRs 54316, 53400, 55933, 55786, 55794). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Add Amazon NLB support

**What this PR does / why we need it**:

This adds support for AWS's NLB for `LoadBalancer` services.

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #

Fixes #52173

**Special notes for your reviewer**:

This is NOT yet ready for merge, but I'd love any feedback before it is.

This requires at least `v1.10.40` of the [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go), which is not yet included in Kubernetes. Per @justinsb, I'm waiting on possibly #48314 to update to `v1.10.40`  or some other PR. 

I tried to make the change as easy to review as possible, so some LoadBalancer logic is duplicated in the `if isNLB(annotations)` blocks. I can refactor that and sprinkle more `isNLB()` switches around, but it might be harder to view the diff.

**Other Notes:**

* NLB's subnets cannot be modified after creation (maybe look for public subnets in all AZ's?).  Currently, I'm just using `c.findELBSubnets()`
* Health check uses TCP with all the NLB default values. I was thinking HTTP health checks via annotation could be added later. Should that go into this PR?
* ~~`externalTrafficPolicy`/`healthCheckNodePort` are ignored. Should those be implemented for this PR?~~
* `externalTrafficPolicy` and subsequent `healthCheckNodePort` are handled properly. This may come with uneven load balancing, as NLB doesn't support weighted backends.
* With classic ELB, you have a security group the ELB is inside of to associate Instance (k8s node) SG rules with a LoadBalancer (k8s Service), but NLB's don't have a security group. Instead, I use the `Description` field on [`ec2.IpRange`](https://docs.aws.amazon.com/sdk-for-go/api/service/ec2/#IpRange) with the following annotations. Is this ok? I couldn't think of another way to associate SG rule to the NLB
    * Node SG gets an rule added for VPC cidr on NodePort for Health Check with annotation in description `kubernetes.io/rule/nlb/health=<loadBalancerName>`
    * Node SG gets an rule added for `loadBalancerSourceRanges` to  NodePort for client traffic with annotation in description `kubernetes.io/rule/nlb/client=<loadBalancerName>`
    * **Note: if `loadBalancerSourceRanges` is unspecified, this opens instance security groups to traffic from `0.0.0.0/0` on the service's nodePorts**
* Respects internal annotation
* Creates a TargetGroup per frontend port: simplifies updates when you have same backend port for multiple front end ports.
* Does not (yet) verify that we're under the NLB limits in terms of # of listeners
* `UpdateLoadBalancer()`  basically just calls `EnsureLoadBalancer` for NLB's. Is this ok?

**Areas for future improvement or optimization**:

* A new annotation indicating a new security group should be created for NLB traffic and instances would be placed in this new SG. (Could bump up against the default limit of 5 SG's per instance)
* Only create a client health check security group rule when the VPC cidr is not a subset of `spec.loadBalancerSourceRanges`
* Consolidate TargetGroups if a service has 2+ frontend ports and the same nodePort.
* A new annotation for specifying TargetGroup Health Check options.

**Release note**:

```release-notes
Add Amazon NLB support - Fixes #52173
```

ping @justinsb @bchav
2017-11-21 15:04:25 -08:00
Kubernetes Submit Queue
d1e711a6af
Merge pull request #55307 from xiangpengzhao/fix-aws-panic
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Check if SleepDelay of AWS request is nil before sign.

**What this PR does / why we need it**:

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #55309

**Special notes for your reviewer**:
/cc @justinsb 

**Release note**:

```release-note
NONE
```
2017-11-21 06:47:30 -08:00
dimitar
8cf7c5e34a allow ELB HC configuration via Service annotations
The constants which have been used so far have been set as default in
case the annotations have not been set.
2017-11-21 14:34:05 +00:00
Kubernetes Submit Queue
5353d588b6
Merge pull request #55611 from stewart-yu/regexMatch
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

using Regexp Match 

**What this PR does / why we need it**:
using regexp match achieve find efficiently

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #

**Special notes for your reviewer**:

**Release note**:

```release-note
NONE
```
2017-11-21 05:58:23 -08:00
Micah Hausler
502e8f8c29 Add aws elbv2 to vendor 2017-11-20 08:52:57 -05:00
Micah Hausler
f9445b9dc7 Add Amazon NLB support 2017-11-20 08:52:57 -05:00
xiangpengzhao
1d6fea99a7 Check if SleepDelay of AWS request is nil before sign. 2017-11-19 16:33:56 +08:00
Kubernetes Submit Queue
cf5d4011ac
Merge pull request #55731 from georgebuckerfield/elb-tagging
Automatic merge from submit-queue (batch tested with PRs 50457, 55558, 53483, 55731, 52842). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Ensure new tags are created on existing ELBs

**What this PR does / why we need it**:

When editing an existing service of type LoadBalancer in an AWS environment and adding the `service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags` annotation, you would expect the new tags to be set on the load balancer, however this doesn't happen currently. The annotation only takes effect if specified when the service is created.

This PR adds an AddTags method to the ELB interface and uses this to ensure tags set in the annotation are present on the ELB. If the tag key is already present, the value will be updated.

This PR does not remove tags that have been removed from the annotation, it only add/updates tags.

**Which issue(s) this PR fixes**:
Fixes #54642 

**Special notes for your reviewer**:
The change requires that the IAM policy of the master instance(s) has the `elasticloadbalancing:AddTags` permission.

**Release note**:

```release-note
Ensure additional resource tags are set/updated AWS load balancers
```
2017-11-18 11:36:22 -08:00
Kubernetes Submit Queue
1b3d880725
Merge pull request #55558 from gnufied/implement-node-taint-for-attaching-volumes
Automatic merge from submit-queue (batch tested with PRs 50457, 55558, 53483, 55731, 52842). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Apply taint when a volume is stuck in attaching state

When a volume is stuck in attaching state for too long on a node, it is best to make node unschedulable so as any other pod may not be scheduled on it.

Fixes https://github.com/kubernetes/kubernetes/issues/55502 

```release-note
AWS: Apply taint to a node if volumes being attached to it are stuck in attaching state
```
2017-11-18 11:36:16 -08:00
Kubernetes Submit Queue
b223955c06
Merge pull request #54507 from micahhausler/aws-elb-security-policy
Automatic merge from submit-queue (batch tested with PRs 54134, 54507). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Added service annotation for AWS ELB SSL policy

**What this PR does / why we need it**:

This work adds a new supported service annotation for AWS clusters, `service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy`, which lets users specify which [predefined AWS SSL policy](http://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-security-policy-table.html) they would like to use.

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #

Fixes #43744

**Special notes for your reviewer**:

While this PR doesn't allow users to define their own cipher policy in an annotation, a user could (out of band) create their own policy on an ELB with the naming convention `k8s-SSLNegotiationPolicy-<my-policy-name>` and specify it with the above annotation.

This is my second k8s PR, and I don't have experience with an e2e test, would that be required for this change? I did run this in a kubeadm cluster and it worked like a charm. I was able to choose different predefined policies, and revert to the default policy when I removed the annotation.

**Release note**:

```release-note
Added service annotation for AWS ELB SSL policy
```
2017-11-17 01:17:11 -08:00
Hemant Kumar
5297c146c1 Fix dangling attach errors
Detach volumes from shutdown nodes and ensure that
dangling volumes are handled correctly in AWS
2017-11-16 08:43:36 -05:00
georgebuckerfield
36d59704d9 Modify the AWS cloud provider to ensure additional load balancer tags are added to existing load balancers 2017-11-14 21:47:50 +00:00
stewart-yu
1a11a1886f using regexp match achieve find efficiently 2017-11-14 12:56:56 +08:00
Hemant Kumar
063cd2dfa5 Apply taint when a volume is stuck in attaching state on node 2017-11-13 12:30:52 -05:00
xiangpengzhao
b44e4b4f86 Raise log level to avoid log spam 2017-11-07 15:19:20 +08:00
Hemant Kumar
c00b136c74 Check for available volume before attach/delete in EBS
We should check for available volume before performing
attach or delete of EBS volume. This will make sure that
we do not blow up API quota of mutable operations in AWS and stay a
good citizen.
2017-11-02 13:36:21 -04:00
Micah Hausler
cb38293093 Added service annotation for AWS ELB SSL policy 2017-10-26 15:21:14 -04:00
FengyunPan
462087fd74 Implement InstanceExistsByProviderID() for cloud providers
Fix #51406
If cloud providers(like aws, gce etc...) implement ExternalID()
and support getting instance by ProviderID , they also implement
InstanceExistsByProviderID().
2017-10-20 14:59:28 +08:00
Jeff Grafton
aee5f457db update BUILD files 2017-10-15 18:18:13 -07:00
Kubernetes Submit Queue
b4dd4edfae Merge pull request #52656 from rrati/aws-fake
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Move AWS Fake implementations out of test 

The AWS fake implementations are in a test file and can't be imported into any other tests.  This makes integration testing difficult.  This PR moves the fake implementations such that they can be used by other entities.

@kubernetes/sig-aws-misc @justinsb
2017-10-02 22:35:33 -07:00
Henrik Schmidt
1339e4cffc Use custom error for "unimplemented" 2017-09-26 09:21:53 +02:00
Kubernetes Submit Queue
2b594750e7 Merge pull request #48145 from imcsk8/loadbalancer-logs
Automatic merge from submit-queue (batch tested with PRs 52240, 48145, 52220, 51698, 51777). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>..

Avoid printing node list for LoadBalancer in log file

**What this PR does / why we need it**:  Production log files get saturated with EnsureLoadBalancer messages, this is problematic for sysadmins. 
This patch avoids printing the node list on the AWS logs so the log file is  more readable.
2017-09-23 09:13:57 -07:00