Automatic merge from submit-queue
Allow conformance tests to run on non-GCE providers
fixes https://github.com/kubernetes/kubernetes/issues/26869
Creates a skeleton provider which has all the required function stubs -- but will allow a previously set "skeleton" KUBERNETES_PROVIDER to not be overriden with "gce".
Automatic merge from submit-queue
Get GCI images from image family
The staging images are now created with image families, so we can get rid of the
image indices stored in GCS. Also, update the GCS bucket used for staging
images.
@Amey-D @spxtr Can you review?
cc/ @kubernetes/goog-image
Automatic merge from submit-queue
Fix hack/update-api-reference-docs on master and (future) branches
- update toplevel api object pattern in swagger doc script: the format of the `register.go` file was changed in e3af3451c8. This fixes the bleeding.
- make `hack/update-api-reference-docs.sh` independent from master during **container run**. Container build is still dependent on master, but container rebuilds never happen I guess on a branch.
A rebuild of `gcr.io/google_containers/gen-swagger-docs:v6` is necessary, plus a cherry-pick onto the v1.3 branch.
Fixes the symptoms of #27685 and makes sure future branches do not break again.
The staging images are now created with image families, so we can get rid of the
image indices stored in GCS. Also, get images based on milestone number instead
of "image type".
Automatic merge from submit-queue
Enable all ppc64le builds, except for hyperkube
Partially fixes: #25886
Talked to @Pensu, and all other binaries seem to work fine
@david-mcmahon @ixdy @Pensu @smarterclayton
Automatic merge from submit-queue
Prep for continuous Docker validation test
```release-note
Add a test config variable to specify desired Docker version to run on GCI.
```
We want to continuously validate Docker releases (#25215), on GCI. This change
adds a new test config variable, `KUBE_GCI_DOCKER_VERSION`, through which we can
specify which version of Docker we want to run on the master and nodes. This
change also patches the Jenkins e2e-runner with the ability to fetch the latest
Docker (pre)release, and sets the aforementioned variable accordingly.
Tested on my local Jenkins instance that was able to start a cluster with the latest Docker version (different from installed version) running on both master and nodes.
@dchen1107 Can you review?
cc/ @andyzheng0831 for changes in `cluster/gce/gci/helper.sh`, and @ixdy @spxtr for changes to the Jenkins e2e-runner
cc/ @kubernetes/goog-image
Automatic merge from submit-queue
Revert "Revert "GCI: add support for network plugin""
PR #27027 added the network plugin support in GCI config, but later a bug in the network plugin broke e2e tests (see issue #27118). The bug was fixed by #27141 and we have been repeatedly run the serial e2e tests more than 10 times to verify the fix. Now it should be safe to put the GCI network plugin support back.
We will first merge in the master branch and monitor the Jenkins serial tests for a while and then cherry-pick it into release-1.3 branch.
Automatic merge from submit-queue
Avoid duplicate building in Jenkins unit/integration job
Partially adopts #26392: don't run `hack/build-go.sh` in the unit/integration job, since we do that already for e2e.
We do need to still build kubectl, however, so do that in `hack/test-cmd.sh`.
x-ref #25940
Automatic merge from submit-queue
Fixes and improvements to Photon Controller backend for kube-up
- Improve reliability of network address detection by using MAC
address. VMware has a MAC OUI that reliably distinguishes the VM's
NICs from the other NICs (like the CBR). This doesn't rely on the
unreliable reporting of the portgroup.
- Persist route changes. We configure routes on the master and nodes,
but previously we didn't persist them so they didn't last across
reboots. This persists them in /etc/network/interfaces
- Fix regression that didn't configure auth for kube-apiserver with
Photon Controller.
- Reliably run apt-get update: Not doing this can cause apt to fail.
- Remove unused nginx config in salt
- Improve reliability of network address detection by using MAC
address. VMware has a MAC OUI that reliably distinguishes the VM's
NICs from the other NICs (like the CBR). This doesn't rely on the
unreliable reporting of the portgroup.
- Persist route changes. We configure routes on the master and nodes,
but previously we didn't persist them so they didn't last across
reboots. This persists them in /etc/network/interfaces
- Fix regression that didn't configure auth for kube-apiserver with
Photon Controller.
- Reliably run apt-get update: Not doing this can cause apt to fail.
- Remove unused nginx config in salt
Automatic merge from submit-queue
federation: fix dns provider initialization issues
This PR is based on the integration test with Google DNS API. This is the first time of full integration test.
So multiple issues was found and I combined all of them in this single PR
1. add dns provider initialization and add ensureDns call when removing federation service.
2. add new flags federation-name and zone-name to controller manager, both are used as part of the dns record name
3. fix assertion failure at rrsets.go#L61, which will cause panic
4. change getFederationDNSZoneName to get zoneName from config instead of hard code
5. change logic of ensureDnsRrsets, only add new dns record when endpointReachable(set to true when ready address is catched) is true
6. fix bug in processEndpointUpdate, only call ensuredns when ready address is caught
7. change behavior of syncService, there is cases that endpoint is created before ingress IP assignment, so before there is defect for this case, ensureDns was not called when service being updated, so if Ingress IP is assigned after endpoint ready address is caught, the corresponding A records can not be created
8. add a checking before update federation service
@nikhiljindal , can you help to add 1.3 milestone when @quinton-hoole is on leave?
Thanks.
[]()
Added a new environment variable API_HOST_IP. API_HOST_IP allows the
user to specify an IPv6 address that is parsable by Golang. API_HOST on
the other hand allows the user to specify the IPv6 address to be used in
a URL's format as described in RFC2732.
Example:
API_HOST_IP="FEDC:BA98:7654:3210:FEDC:BA98:7654:3210"
API_HOST="[FEDC:BA98:7654:3210:FEDC:BA98:7654:3210]
or
API_HOST_IP="FEDC:BA98:7654:3210:FEDC:BA98:7654:3210"
API_HOST="[${API_HOST_IP}]"
Signed-off-by: André Martins <aanm90@gmail.com>
Automatic merge from submit-queue
fix recursive & non-recursive kubectl get of generic output format
This PR fixes the issues with `kubectl get` in https://github.com/kubernetes/kubernetes/issues/26466
Changes made:
- fix printing when using the generic output format in both non-recursive & recurvise settings to ensure that errors are being shown
- add tests to check printing generic output in a **non-recursive** setting with non-existent pods
- clean up the **recursive** `kubectl get` tests
/cc @janetkuo
Automatic merge from submit-queue
Stop checking in generated documents
Part of #26205
Auto generated docs are **NO LONGER CHECKED IN**, only placeholders.
To generate them, e.g. before exporting docs, run hack/generate-docs.sh.
hack/verify-generated-docs.sh ensures that generated docs are merely the
placeholder text.
hack/update-generated-docs.sh puts the placeholder text in the proper
places.
The old munge behavior is moved into hack/{update|verify}-munge-docs.sh.
Pull the latest build of the release the server is running, rather
than matching exact version. This allows GKE to pick up test fixes
from branch head, instead of waiting for a patch.
Automatic merge from submit-queue
Add a custom main instead of the standard test main, to reduce stack …
Adds a custom test main handler (see: `TestMain` in https://golang.org/pkg/testing/ for details)
Partial fix for https://github.com/kubernetes/kubernetes/issues/25965
This does the standard timeout, but strips non-kubernetes stacks out of the stack trace (e.g. it filters things like:
```
goroutine 466 [IO wait, 7 minutes]:
net.runtime_pollWait(0x7fd74c4672c0, 0x72, 0xc821614000)
/usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc8215c21b0, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8215c21b0, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8215c2150, 0xc821614000, 0x1000, 0x1000, 0x0, 0x7fd74c491050, 0xc820014058)
/usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc820a5a090, 0xc821614000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7fd74c465258, 0xc820a5a090, 0xc8215f0068, 0xc821614000, 0x1000, 0x1000, 0x405773, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:1687 +0x67
net/http.(*noteEOFReader).Read(0xc8215ae1a0, 0xc821614000, 0x1000, 0x1000, 0xc82159ad1d, 0x0, 0x0)
<autogenerated>:284 +0xd0
bufio.(*Reader).fill(0xc8202a2b40)
/usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc8202a2b40, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc8215f0000)
/usr/local/go/src/net/http/transport.go:1073 +0x177
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:857 +0x10a6
```
We may want to get even more aggressive in the future.
@kubernetes/sig-testing
Automatic merge from submit-queue
GCI: add support for network plugin
I had run e2e against a cluster with both master and nodes on GCI a couple of times. The PR auto tests will cover the hybrid cluster with just master on GCI.
cc/ @roberthbailey @fabioy @kubernetes/goog-image
Auto generated docs are **NO LONGER CHECKED IN**, only placeholders.
To generate them, e.g. before exporting docs, run hack/generate-docs.sh.
hack/verify-generated-docs.sh ensures that generated docs are merely the
placeholder text.
hack/update-generated-docs.sh puts the placeholder text in the proper
places.
The old munge behavior is moved into hack/{update|verify}-munge-docs.sh.
Also includes other improvements:
- Makefile rule to run tests against remote instance using existing host or image
- Makefile will reuse an instance created from an image if it was not torn down
- Runner starts gce instances in parallel with building source
- Runner uses instance ip instead of hostname so that it doesn't need to resolve
- Runner supports cleaning up files and processes on an instance without stopping / deleting it
- Runner runs tests using `ginkgo` binary to support running tests in parallel
Automatic merge from submit-queue
Enable support for memory eviction configuration via salt
Added evictions based on memory by default whenever the available memory is < 100Mi.
Updated GCE and GCI.
Automatic merge from submit-queue
Always skew the client when running version-skew tests
As mentioned in the comments:
- for upgrade jobs, we want kubectl to be at the same version as master.
- for client skew tests, we want to use the skewed kubectl (that's what we're testing).
So, remove gate on `JENKINS_USE_SKEW_KUBECTL`, and always use the skewed `kubectl`.
This should go in before https://github.com/kubernetes/test-infra/pull/109.
We want to continuously validate Docker releases (#25215), on GCI. This change
adds a new test config variable, `KUBE_GCI_DOCKER_VERSION`, through which we can
specify which version of Docker we want to run on the master and nodes. This
change also patches the Jenkins e2e-runner with the ability to fetch the latest
Docker (pre)release, and sets the aforementioned variable accordingly.
Automatic merge from submit-queue
Don't allow deps with no discernible license
This updates the few deps we had with no LICENSE file to current versions that do have that file. It also disallows new deps without obvious licenses.
Automatic merge from submit-queue
Attach/Detach Controller Kubelet Changes
This PR contains changes to enable attach/detach controller proposed in #20262.
Specifically it:
* Introduces a new `enable-controller-attach-detach` kubelet flag to enable control by attach/detach controller. Default enabled.
* Removes all references `SafeToDetach` annotation from controller.
* Adds the new `VolumesInUse` field to the Node Status API object.
* Modifies the controller to use `VolumesInUse` instead of `SafeToDetach` annotation to gate detachment.
* Modifies kubelet to set `VolumesInUse` before Mount and after Unmount.
* There is a bug in the `node-problem-detector` binary that causes `VolumesInUse` to get reset to nil every 30 seconds. Issue https://github.com/kubernetes/node-problem-detector/issues/9#issuecomment-221770924 opened to fix that.
* There is a bug here in the mount/unmount code that prevents resetting `VolumeInUse in some cases, this will be fixed by mount/unmount refactor.
* Have controller process detaches before attaches so that volumes referenced by pods that are rescheduled to a different node are detached first.
* Fix misc bugs in controller.
* Modify GCE attacher to: remove retries, remove mutex, and not fail if volume is already attached or already detached.
Fixes#14642, #19953
```release-note
Kubernetes v1.3 introduces a new Attach/Detach Controller. This controller manages attaching and detaching volumes on-behalf of nodes that have the "volumes.kubernetes.io/controller-managed-attach-detach" annotation.
A kubelet flag, "enable-controller-attach-detach" (default true), controls whether a node sets the "controller-managed-attach-detach" or not.
```
Automatic merge from submit-queue
Fixes#26526 - hack/update-api-reference-docs.sh
I opened a Pull request to fix this issue https://github.com/kubernetes/kubernetes/issues/26526
The problem is that the update script ignores white spaces but the verify script doesn't which leads to a strange behaviour -> you use the update script but the verify script tells you that the api docs are not up to date.
This PR contains Kubelet changes to enable attach/detach controller control.
* It introduces a new "enable-controller-attach-detach" kubelet flag to
enable control by controller. Default enabled.
* It removes all references "SafeToDetach" annoation from controller.
* It adds the new VolumesInUse field to the Node Status API object.
* It modifies the controller to use VolumesInUse instead of SafeToDetach
annotation to gate detachment.
* There is a bug in node-problem-detector that causes VolumesInUse to
get reset every 30 seconds. Issue https://github.com/kubernetes/node-problem-detector/issues/9
opened to fix that.
Add it as a special case package root and import the license file. This was
the only UNKNOWN license, prior to the change to not look at upstream repo
state.
This change recovers some of the side effects of
https://github.com/kubernetes/kubernetes/pull/26197, i.e., keeps the defaults of
`NODE_IMAGE` and `NODE_IMAGE_PROJECT` to `MASTER_IMAGE` and
`MASTER_IMAGE_PROJECT`, for backward compatibility. Although it keeps
`OS_DISTRIBUTION` defaulting to `gci`, the default settings of these vars are
moved to `cluster/gce/util.sh` and conditioned on `OS_DISTRIBUTION==gci`.
Automatic merge from submit-queue
Don't set the env var CC when not cross-compiling
I noticed that this script was trying to use `arm-linux-gnueabi-gcc` also when running natively on arm.
When running natively, `CC` should always be `gcc` (which also is the default)
Also added `federation-controller-manager` to the static list, I think someone forgot to do that.
@ixdy @david-mcmahon @spiffxp @spxtr
Automatic merge from submit-queue
Get install-etcd.sh to work on OSX
Check the platform before we download the etcd binary. On OSX, save
the zip file, unzip it and link the directory with the correct
name.
Fixes#26041
Automatic merge from submit-queue
Add -addr option to hack/lib/etcd.h
Added -addr to etcd hack script so it listens on the same address specified in the `host` variable.
Signed-off-by: André Martins <aanm90@gmail.com>
Automatic merge from submit-queue
Support per-test-environment ginkgo flags for node e2e tests to facilitate skipping miss behaving tests in PR builder
We had an issue today where some node e2e tests were timing out in the pr builder. We want to be able to skip tests in the pr builder and leave them running in the CI if this happens again.
[]()
Automatic merge from submit-queue
Adds --network-plugin-dir argument to hack/local-up-cluster.sh
CNI Network Plugin developers who want to use hack/local-up-cluster.sh need to specify the --network-plugin-dir option to kubelet.
Automatic merge from submit-queue
Add more global options to local-up-cluster.sh
I added a couple more global variables to the `local-up-cluster.sh` script.
The `KUBELET_HOST` environment name make sense to me. If it isn't the best name I will change it.
The `ETCD` ones make sense since they are being used under `hack/lib/etcd.sh`
Signed-off-by: André Martins <aanm90@gmail.com>
Automatic merge from submit-queue
kube-controller-manager: Add configure-cloud-routes option
This allows kube-controller-manager to allocate CIDRs to nodes (with
allocate-node-cidrs=true), but will not try to configure them on the
cloud provider, even if the cloud provider supports Routes.
The default is configure-cloud-routes=true, and it will only try to
configure routes if allocate-node-cidrs is also configured, so the
default behaviour is unchanged.
This is useful because on AWS the cloud provider configures routes by
setting up VPC routing table entries, but there is a limit of 50
entries. So setting configure-cloud-routes on AWS would allow us to
continue to allocate node CIDRs as today, but replace the VPC
route-table mechanism with something not limited to 50 nodes.
We can't just turn off the cloud-provider entirely because it also
controls other things - node discovery, load balancer creation etc.
Fix#25602
Automatic merge from submit-queue
Push public key to project if not already present
Fixes https://github.com/kubernetes/kubernetes/issues/26128
Fixes https://github.com/kubernetes/kubernetes/issues/26129
Whenever the ssh key changes (which happens right now whenever we add/change a new jenkins agent) this key will not get pushed to the project until either `gcloud compute ssh` or `gcloud compute config-ssh` runs. As a result instances on this project will reject ssh attempts with this key.
In many cases this will not happen until after a test on a project fails and we attempt to ssh to the nodes to grab logs.
This verifies the presence of the ssh key before starting tests, and attempts to add it if it is missing.
Jenkins relies on junit.xml to identify test failures
and non-0 exit codes to indentify infrastructure failures.
Test failures in kubemark tests should not cause the test
script to exit non-0. Infrastructure failures should.
- Add function to dump cluster logs without exiting (refactor)
- Change `test/kubemark/stop-kubemark.sh` to be run regardless of whether tests fail or not
- Exit code for failed tests overwritten to be the exit code of dumping the cluster logs
This code used to actually reach out to the internet to look for files. This
is flaky, slow, and semantically WRONG. The license that is upstream might
actually be different than what we have vendored. Only look at local files.
This now passes back-to-back updates and verifies.
This allows kube-controller-manager to allocate CIDRs to nodes (with
allocate-node-cidrs=true), but will not try to configure them on the
cloud provider, even if the cloud provider supports Routes.
The default is configure-cloud-routes=true, and it will only try to
configure routes if allocate-node-cidrs is also configured, so the
default behaviour is unchanged.
This is useful because on AWS the cloud provider configures routes by
setting up VPC routing table entries, but there is a limit of 50
entries. So setting configure-cloud-routes on AWS would allow us to
continue to allocate node CIDRs as today, but replace the VPC
route-table mechanism with something not limited to 50 nodes.
We can't just turn off the cloud-provider entirely because it also
controls other things - node discovery, load balancer creation etc.
Fix#25602
Automatic merge from submit-queue
Add release_1_3 clientset in update-codegen
Add release_1_3 clientset in update-codegen to keep it update-to-date; update the generated clientset.
Automatic merge from submit-queue
Attach Detach Controller Business Logic
This PR adds the meat of the attach/detach controller proposed in #20262.
The PR splits the in-memory cache into a desired and actual state of the world.
Automatic merge from submit-queue
use kubectl-with-retry on pause & resume
attempts to fix https://github.com/kubernetes/kubernetes/issues/25645 by using `kubectl-with-retry` on `rollout {pause,resume}` (`resume` is for safe measures) instead of `kubectl` directly, as is done with other `rollout {pause,resume}` tests in this same script.
Automatic merge from submit-queue
e2e-runner: Drop Trusty support in favor of GCI
Now that GCI (Google Container-VM Image) is out, we will start running e2e tests
with it instead of Ubuntu Trusty. This change updates `e2e-runner.sh` to replace
the Trusty related logic with GCI.
_Note that this change has to go in the same time as https://github.com/kubernetes/test-infra/pull/54_
@spxtr Can you review?
cc/ @andyzheng0831 @kubernetes/goog-image
Split controller cache into actual and desired state of world.
Controller will only operate on volumes scheduled to nodes that
have the "volumes.kubernetes.io/controller-managed-attach" annotation.
- Add junit test reported
- Write etcd.log, kubelet.log and kube-apiserver.log to files instead of stdout
- Scp artifacts to the jenkins WORKSPACE
Fixes#25966
Automatic merge from submit-queue
Renaming federated-apiserver to federation-apiserver
Its not a "federated" apiserver. Its an apiserver for federation.
The corresponding controller manager is also called federation-controller-manager.
If we are going to rename, it should be done in 1.3 before people write scripts referencing the binary name.
Automatic merge from submit-queue
Add a 'kubectl clusterinfo dump' option
Ref: #3500
@bgrant0607 @smarterclayton @jszczepkowski
Usage:
```
# Dump current cluster state to stdout
kubectl clusterinfo dump
# Dump current cluster state to /tmp
kubectl clusterinfo dump --output-directory=/tmp
# Dump all namespaces to stdout
kubectl clusterinfo dump --all-namespaces
# Dump a set of namespaces to /tmp
kubectl clusterinfo dump --namespaces default,kube-system --output-directory=/tmp
```
<!-- Reviewable:start -->
---
This change is [<img src="http://reviewable.k8s.io/review_button.svg" height="35" align="absmiddle" alt="Reviewable"/>](http://reviewable.k8s.io/reviews/kubernetes/kubernetes/20672)
<!-- Reviewable:end -->
Automatic merge from submit-queue
Improve the speed of do-nothing build.
As @thockin found out here https://github.com/kubernetes/kubernetes/issues/24518, vast majority of the do-nothing build time is spent in rebuilding the test binaries. There is no staleness check support for test binaries.
This commit implements the staleness checks for test binaries and uses them while building packages.
On my workstation, do-nothing hack/build-go.sh time goes from ~20 secs to ~4 secs, of which only ~1 sec is from doing test binary staleness check now (as opposed to ~17 secs it took to build the test binaries before). I did some experiments to bring this time down to <1 sec. I measured using go test -bench, but it was not very useful in this case. I believe, a vast majority of that ~1 second is being spent in fork/exec and piping the results back to the staleness check program along with the ser-deser involved, but it needs to be validated. Not a proof, but to provide some supporting evidence to this claim, running `go list -f format packages` in the shell takes about 600ms irrespective of what's in the format.
Tests are TBD. I am still trying to figure out how to test this, but I would like to get early feedback
cc @mikedanese @mml
