Automatic merge from submit-queue
Add quick-verify make rule.
This is useful for humans to run to catch obvious problems before
pushing commits and waiting for CI to run verify checks.
Quick mode only runs a whitelist of verify scripts that are reasonably fast.
I set the initial bar arbitrarily at <10s each on my workstation.
The whole set runs in <30s for me, assuming I had already run `make` and
`hack/godep-restore.sh`. This is compared to the full `make verify`
which takes [I don't know how long because I gave up after 45min].
Automatic merge from submit-queue (batch tested with PRs 47066, 48892, 48933, 48854, 48894)
azure: msi: add managed identity field, logic
**What this PR does / why we need it**: Enables managed service identity support for the Azure cloudprovider. "Managed Service Identity" allows us to ask the Azure Compute infra to provision an identity for the VM. Users can then retrieve the identity and assign it RBAC permissions to talk to Azure ARM APIs for the purpose of the cloudprovider needs.
Per the commit text:
```
The azure cloudprovider will now use the Managed Service Identity
to retrieve access tokens for the Azure ARM APIs, rather than
requiring hard-coded, user-specified credentials.
```
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: n/a
**Special notes for your reviewer**: none
**Release note**:
```release-note
azure: support retrieving access tokens via managed identity extension
```
cc: @brendandburns @jdumars @anhowe
Automatic merge from submit-queue
Azure PD (Managed/Blob)
This is exactly the same code as this [PR](https://github.com/kubernetes/kubernetes/pull/41950). It has a clean set of generated items. We created a separate PR to accelerate the accept/merge the PR
CC @colemickens
CC @brendandburns
**What this PR does / why we need it**:
1. Adds K8S support for Azure Managed Disks.
2. Adds support for dedicated blob disks (1:1 to storage account) in addition to shared blob disks (n:1 to storage account).
3. Automatically manages the underlying storage accounts. New storage accounts are created at 50% utilization. Max is 100 disks, 60 disks per storage account.
2. Addresses the current issues with Blob Disks:
..* Significantly faster attach process. Disks are now usually available for pods on nodes under 30 sec if formatted, under a min if not formatted.
..* Adds support to move disks between nodes.
..* Adds consistent attach/detach behavior, checks if the disk is leased/attached on a different node before attempting to attach to target nodes.
..* Fixes a random hang behavior on Azure VMs during mount/format (for both blob + managed disks).
..* Fixes a potential conflict by avoiding the use of disk names for mount paths. The new plugin uses hashed disk uri for mount path.
The existing AzureDisk is used as is. Additional "kind" property was added allowing the user to decide if the pd will be shared, dedicated or managed (Azure Managed Disks are used).
Due to the change in mounting paths, existing PDs need to be recreated as PV or PVCs on the new plugin.
Automatic merge from submit-queue (batch tested with PRs 48864, 48651, 47703)
Enable logexporter mechanism to dump logs from k8s nodes to GCS directly
Ref https://github.com/kubernetes/kubernetes/issues/48513
This adds support for logexporter from k8s side. Next I'll send a PR adding support from test-infra side.
/cc @kubernetes/sig-scalability-misc @kubernetes/test-infra-maintainers @fejta @wojtek-t @gmarek
Automatic merge from submit-queue
add dockershim checkpoint node e2e test
Add a bunch of disruptive cases to test kubelet/dockershim's checkpoint work flow.
Some steps are quite hacky. Not sure if there is better ways to do things.
This is useful for humans to run to catch obvious problems before
pushing commits and waiting for CI to run verify checks.
Quick mode only runs a whitelist of verify scripts that are reasonably fast.
I set the initial bar arbitrarily at <10s each on my workstation.
The whole set runs in <30s for me, assuming I had already run `make` and
`hack/godep-restore.sh`. This is compared to the full `make verify`
which takes [I don't know how long because I gave up after 45min].
Automatic merge from submit-queue (batch tested with PRs 48594, 47042, 48801, 48641, 48243)
Prepare to introduce websockets for exec and portforward
Refactor the code in remotecommand to better represent the structure of
what is common between portforward and exec.
Ref #48633
Automatic merge from submit-queue (batch tested with PRs 48279, 48566, 48319, 48794, 47952)
Removed mesos flags from known-flags.txt.
**What this PR does / why we need it**:
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
none
```
Automatic merge from submit-queue (batch tested with PRs 48698, 48712, 48516, 48734, 48735)
Name change: s/timstclair/tallclair/
I changed my name, and I'm migrating my user name to be consistent.
Automatic merge from submit-queue (batch tested with PRs 47948, 48631, 48693, 48549, 47593)
add generated clients. modify codegen script
**What this PR does / why we need it**:
Adds in the generated clientsets for the sample apiserver. Modifies the update-codgen script to copy over the client from the generated location. We need it in order to be able to add integrations and unit tests that make use of the clientsets and their fakes.
**Special notes for your reviewer**:
@deads2k @p0lyn0mial hopefully done the correct thing here. Not 100% sure on needing the copy but it seemed what was needed to get the import paths correct?
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue
kube-apiserver: tests for aggregation and CRDs via delegation
In our integration tests we do not use the real kube-apiserver setup code, but mock our own. Here I use the actual `cmd/kube-apiserver/app.Run()` func with an testing etcd server. This can test the whole delegation chain of aggregator, apiextensions and kube-apiserver.
Automatic merge from submit-queue (batch tested with PRs 47232, 48625, 48613, 48567, 39173)
Make kube-proxy's MetricsBindAddress configurable via flag
**What this PR does / why we need it**: From #48600, `MetricsBindAddress` used to be configurable through flag (coupled with `HealthzBindAddress`). But this functionality went away after #44968 separated healthz server from metrics server. This PR adds that functionality back.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*:
**Special notes for your reviewer**:
/assign @ncdc
cc @bowei @r0bj
**Release note**:
```release-note
none
```
Automatic merge from submit-queue (batch tested with PRs 48196, 42783, 48507, 47719, 46138)
fix parse resource in setting selector
**What this PR does / why we need it**:
**Which issue this PR fixes**: fixes#47718
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
fix issue where binaries not being removed. Add new client packages to lint
put new linted packages in correct location. remove flag from variables
Add all pkgs that should be linted and run sort
Add output from hack/update-bazel.sh
remove genclient=true from FischerList and regenerate client
re update bazel build files
fix missing resource method in register.go
Automatic merge from submit-queue (batch tested with PRs 47040, 48597, 48608, 48653)
Fix godep verify to use godep restore script
**What this PR does / why we need it**:
A bug was introduced in #48615. `hack/verify-godeps.sh` only downloads and compares if godeps have changed, so it wasn't caught on the original PR. However, when it does run (e.g. https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/48630/pull-kubernetes-verify/38350/) it fails because the godep-save script now doesn't permit a compex GOPATH. verify-godeps.sh actually sets one because it restores godeps not using the `hack/godep-restore.sh` script.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
/assign @sttts
/priority failing-test
Automatic merge from submit-queue (batch tested with PRs 48497, 48604, 48599, 48560, 48546)
Move pkg/apimachinery/test to apimachinery
Fixes https://github.com/kubernetes/kubernetes/issues/48265
for circular dependency reason, all the test file contain both `"k8s.io/apimachinery/pkg/apis/meta/v1"` and `"k8s.io/apimachinery/pkg/runtime"` is hard to move to a ideal location, so I create a separated test package for those files
I also bump the example package :https://github.com/kubernetes/kubernetes/tree/master/staging/src/k8s.io/apiserver/pkg/apis/example here for apimachinery, @sttts told me it's ok if to bump file if only for test EDIT: seems it's no need to bump, will update soon
```
NONE
```
Automatic merge from submit-queue (batch tested with PRs 47234, 48410, 48514, 48529, 48348)
Remove unused sub-pkgs in pkg/util
**What this PR does / why we need it**:
Remove no longer used sug-pkgs in pkg/util
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#48386
**Special notes for your reviewer**:
/cc @deads2k
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 47435, 46044)
kubeadm: Make self-hosting work and split it out to a phase
**What this PR does / why we need it**:
- Removes the old self-hosting code
- Puts the new self-hosting code in `phases/selfhosting`
- The new code reads manifests from disk (static pods)...
- ...mutates the PodSpec as necessary...
- ...and posts the DaemonSet to the API Server...
- ...and waits for it to come up
- Uses DaemonSets for all control plane components
- Creates a `kubeadm alpha phase selfhosting` command that can be invoked against any kubeadm-cluster after install.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
fixes: https://github.com/kubernetes/kubeadm/issues/127
(large part of at least)
**Special notes for your reviewer**:
Please only review the fourth commit, based on https://github.com/kubernetes/kubernetes/pull/47345
**Release note**:
```release-note
kubeadm: Make self-hosting work by using DaemonSets and split it out to a phase that can be invoked via the CLI
```
@kubernetes/sig-cluster-lifecycle-pr-reviews @jbeda
Automatic merge from submit-queue (batch tested with PRs 48309, 48489)
Check if golint exists first in hack/verify-golint.sh
**What this PR does / why we need it**:
Check if golint exists first in hack/verify-golint.sh
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#48488
**Special notes for your reviewer**:
nope.
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 48480, 48353)
remove tpr api access
xref https://github.com/kubernetes/kubernetes/issues/48152
TPR tentacles go pretty deep. This gets us started by removing API access and we'll move down from there.
@kubernetes/sig-api-machinery-misc
@ironcladlou this should free up the GC implementation since TPRs will no longer be present and failing.
```release-note
Removing TPR api access per https://github.com/kubernetes/kubernetes/issues/48152
```
Automatic merge from submit-queue (batch tested with PRs 47918, 47964, 48151, 47881, 48299)
Add ApiEndpoint support to GCE config.
**What this PR does / why we need it**:
Add the ability to change ApiEndpoint for GCE.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
None
```
Automatic merge from submit-queue (batch tested with PRs 48295, 48298, 47339, 44910, 48037)
Make the `--controllers` flag configurable in hack/local-up-cluster.sh
**What this PR does / why we need it**:
add options to enable tokencleaner,bootstrapsigner controller for bootstrap token testing
**Release note**:
```
None
```
Automatic merge from submit-queue (batch tested with PRs 48295, 48298, 47339, 44910, 48037)
Make Makefiles in `test/images/` compatible with multiple architectures
**What this PR does / why we need it**:
This PR is for making test images multi architecture for different platforms like amd64, arm, arm64, ppc64le
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #31331
**Special notes for your reviewer**:
- Actual tests need to be modified to use these images based on the architecture later.
- Not covering the cross building of docker images for `s390x` platform due to problem faced while running containers with `qemu-s390x-static`
- Will submit separate PR for `volume and pet` test images
- This PR depends on - https://github.com/kubernetes/ingress/pull/587
**Release note**:
```NONE```
Automatic merge from submit-queue (batch tested with PRs 46850, 47984)
Enable vmodule support for all components
Support vmodule (via `LOG_SPEC`) for all local cluster components.
```release-note
NONE
```
When $GO_OUT is not set, line 152 will output the error:
[: ==: unary operator expected. This occurs because the if condition becomes
if [ == "" ]. This results in an error because == is a binary operator.
Automatic merge from submit-queue (batch tested with PRs 47650, 47936, 47939, 47986, 48006)
External dependency of k8s.io/api
Fix https://github.com/kubernetes/kubernetes/issues/48007
It's unfortunate that k8s.io/api has external dependencies.
Most of the dependencies are introduced by "k8s.io/apimachinery/pkg/util/intstr" and ugorji.
Automatic merge from submit-queue
correct the name of the script to generate swagger doc
**What this PR does / why we need it**: The name of the script to generate swagger doc is not correct, this PR is to fix it.
**Which issue this PR fixes**: fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 47921, 45984, 46829, 46896, 46923)
Remove unnecessary wrapper flags
**What this PR does / why we need it**:
Drop KUBE_GOFLAGS, KUBE_GOGCFLAGS, KUBE_GOLDFLAGS references
from the build infrastructure
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#47296
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 47921, 45984, 46829, 46896, 46923)
Create output_dir if doesn't exist
**What this PR does / why we need it**:
Minor fix to the swagger spec gen, that if the directory doesn't exist, create it before copying.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 47921, 45984, 46829, 46896, 46923)
Move govet verify into rest of verify*-.sh scripts
**What this PR does / why we need it**:
Instead of having two govet scripts, consolidate them to into one and have both the Makefile and verify.sh scripts target the same script. This also will allow proper syntax highlighting and timing when the vet script is run as part of `make verify`.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes https://github.com/kubernetes/test-infra/issues/2725
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
/cc @fejta @rmmh
Automatic merge from submit-queue
Give developer more help info when running integration tests without etcd found.
**What this PR does / why we need it**:
When running integration tests, if etcd version on env is lower than expected, developer will get info `You can use 'hack/install-etcd.sh' to install a copy in third_party/.` It would be better to have the same info if no etcd installed on env.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Don't revert, change boundingdirs in Makefile for deepcopy-gen
manually fix pkg/client/clientset_generated/clientset/typed/core/v1/pod_expansion.go because external policy types are moved now
Automatic merge from submit-queue
Regenerate docs (if necessary) during cherry-pick operations.
This change ensures that cherrypicks are coupled with any related doc generation required.
closes#44533
This commit ensures that:
- go vet will be run as part of the make verify target
- the vet make-rule script won't be run directly, as generated_files won't be run in that case
- that go vet errors show up in the build log with a start time, finish time, and SUCCESS/FAILED message as part of the verify make rule script
Automatic merge from submit-queue
Fixed the issue with formatting issues not being reported when verification fails
**What this PR does / why we need it**:
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
If there are 2 or more PR(s) in the queue, they will end up with
conflicts (and rechecks). So let us remove the timestamp entirely
when we generate the files.
Fixes#46814
Automatic merge from submit-queue (batch tested with PRs 46327, 47166)
mark --network-plugin-dir deprecated for kubelet
**What this PR does / why we need it**:
**Which issue this PR fixes** : fixes#43967
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 47530, 47679)
Use cos-stable-59-9460-64-0 instead of cos-beta-59-9460-20-0.
Remove dead code that has now moved to another repo as part of #47467
**Release note**:
```release-note
NONE
```
/sig node
Automatic merge from submit-queue
clientgo/examples: add ToC for examples
Also add authenticate- prefix to auth samples. This patch could use some
improvement explaining workqueue and TPR examples as I'm not entirely sure.
/assign @caesarxuchao
Signed-off-by: Ahmet Alp Balkan <ahmetb@google.com>
- It contains a fix for ipaliasing.
- It contains a fix which decouples GPU driver installation from kernel
version.
Remove dead code that has now moved to another repo as part of #47467
Automatic merge from submit-queue
fix --local flag for kubectl commands
Fixes https://github.com/kubernetes/kubernetes/issues/47079
**Release note**:
```release-note
NONE
```
Fixes the `--local` flag for `kubectl set ...` sub-commands.
**As of the 1.7 release**, `PrinterForCommand` was updated to [use a mapper and typer for unstructured objects](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/cmd/util/factory_builder.go#L52), which further prevented the use of `--local` when there was no connection to an api server.
**before** (with no connection to a server)
```
$ kubectl set resources -f pod.json --limits=cpu=200m,memory=512Mi --local
error: unable to connect to a server to handle "pods": Get https://10.13.137.149:8443/api: dial tcp 10.13.137.149:8443: getsockopt: connection refused
```
**after** (with no connection to a server)
```
$ kubectl set resources -f pod.json --limits=cpu=200m,memory=512Mi --local
NAME READY STATUS RESTARTS AGE
mypod 0/1 0 <unknown>
```
cc @smarterclayton @fabianofranz
Automatic merge from submit-queue (batch tested with PRs 47492, 47542, 46800, 47545, 45764)
Fix TPR watches.
Fixes#47027
TPR watch has been broken since #44350.
Automatic merge from submit-queue (batch tested with PRs 47204, 46808, 47432, 47400, 47099)
support junit in test-cmd
Output junit result for cmd tests.
Fixes#45196
There will be a followup PR to refactor the cmd test into functions as test cases there is not a lot changes in `hack/make-rules/test-cmd-util.sh`
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 47204, 46808, 47432, 47400, 47099)
Make kubectl apply add change-cause before patching
**What this PR does / why we need it**: We shouldn't patch the project with applied change, and then patch again with the change cause. Otherwise, DaemonSet change cause wouldn't be copied to its history (after the first patch, history will be created with the old change cause).
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#47210
**Special notes for your reviewer**:
/assign @mengqiy
@kubernetes/sig-apps-bugs @kubernetes/sig-cli-maintainers
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 47470, 47260, 47411, 46852, 46135)
Write reports for each upgrade test
Due to the way Ginkgo runs individual test cases and the level of coordination required for the upgrade tests, they were all run under a single Ginkgo test case. This PR generates and auxiliary report that break out the results of each upgrade test. This is accomplished by:
1) Wrapping `ginkgo.Fail` and `ginkgo.Skip` to get the actual failure or skip messages.
2) Recovering that info in the upgrade test to generate an auxiliary report.
I suggest reviewing commit by commit.
Sample report: https://storage.googleapis.com/krouseytestreports/logs/results/1/artifacts/junit_upgrades.xmlFixes: #47371
Automatic merge from submit-queue (batch tested with PRs 46441, 43987, 46921, 46823, 47276)
Azure plugin for client auth
This is an Azure Active Directory plugin for client authentification. It provides an integration with Azure CLI 2.0 login command. It can also be used standalone, in that case it will use the device code flow to acquire an access token.
More details are provided in the README.md file.
https://github.com/kubernetes/kubectl/issues/29
cc @brendandburns @colemickens
Automatic merge from submit-queue
Change what is stored in DaemonSet history `.data`
**What this PR does / why we need it**:
In DaemonSet history `.data`, store a strategic merge patch that can be applied to restore a DaemonSet. Only PodSpecTemplate is saved.
This will become consistent with the data stored in StatefulSet history.
Before this fix, a serialized pod template is stored in `.data`; however, seriazlized pod template isn't a `runtime.RawExtension`, and caused problems when controllers try to patch the history's controller ref.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#47008
**Special notes for your reviewer**: @kubernetes/sig-apps-bugs @erictune @kow3ns @kargakis @lukaszo @mengqiy
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue
bazel: stamp multiple packages by using x_defs instead of linkstamp in go_binary rules
**What this PR does / why we need it**: Fixes regression introduced sometime in the last few months that prevented bazel-built clusters from identifying version properly.
It does so by updating the bazelbuild/rules_go and kubernetes/repo-infra dependencies to support using stamp values in `go_binary` `x_defs`, and then changing our `go_binary` rules to use `x_defs` instead of `linkstamp`.
This whole charade is necessary because we need to stamp version information in multiple packages.
This pretty much only affects the bazel build, so it should be low risk.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#45298
**Special notes for your reviewer**: depends on https://github.com/kubernetes/repo-infra/pull/18; should not be merged before it.
**Release note**:
```release-note
NONE
```
/assign @spxtr @mikedanese
Automatic merge from submit-queue (batch tested with PRs 46979, 47078, 47138, 46916)
Add a secretbox and AES-CBC path for encrypt at rest
Add a secretbox and AES-CBC encrypt at rest provider and alter the config, based on feedback from security review. AES-CBC is more well reviewed and generally fits better with common criteria and FIPS, secretbox is newer and faster than CBC.
```release-note
Add secretbox and AES-CBC encryption modes to at rest encryption. AES-CBC is considered superior to AES-GCM because it is resistant to nonce-reuse attacks, and secretbox uses Poly1305 and XSalsa20.
```
Automatic merge from submit-queue (batch tested with PRs 47024, 47050, 47086, 47081, 47013)
apiextensions-apiserver: Fix decoding of DeleteOptions.
Fixes#47072 by making apiextensions-apiserver capable of decoding unversioned DeleteOptions, rather than only handling Unstructured objects (i.e. Custom Resources).
This also closes#46736 and #37554 since the added regression test works for TPR as well.
Automatic merge from submit-queue (batch tested with PRs 47024, 47050, 47086, 47081, 47013)
client-go: deprecate TPR example and add CRD example
/cc @nilebox
Part of https://github.com/kubernetes/kubernetes/issues/46702
Drop KUBE_GOFLAGS, KUBE_GOGCFLAGS, KUBE_GOLDFLAGS references
from the build infrastructure. There are some usages still
for KUBE_GOFLAGS, so we should print a deprecation message
until all those are fixed. If both KUBE_GOFLAGS and GOFLAGS
are just then we just bail out.
Automatic merge from submit-queue (batch tested with PRs 46235, 44786, 46833, 46756, 46669)
implements StatefulSet update
**What this PR does / why we need it**:
1. Implements rolling update for StatefulSets
2. Implements controller history for StatefulSets.
3. Makes StatefulSet status reporting consistent with DaemonSet and ReplicaSet.
https://github.com/kubernetes/features/issues/188
**Special notes for your reviewer**:
**Release note**:
```release-note
Implements rolling update for StatefulSets. Updates can be performed using the RollingUpdate, Paritioned, or OnDelete strategies. OnDelete implements the manual behavior from 1.6. status now tracks
replicas, readyReplicas, currentReplicas, and updatedReplicas. The semantics of replicas is now consistent with DaemonSet and ReplicaSet, and readyReplicas has the semantics that replicas did prior to this release.
```
Automatic merge from submit-queue (batch tested with PRs 46235, 44786, 46833, 46756, 46669)
federation: Add admission controller for policy-based placement
@nikhiljindal
Here's the initial version of the scheduling policy admission controller. It's at the point where it would benefit from having another pair of eyes look at it. The main thing I'm unsure of is the serialization of Kube resources for the webhook/query call.
Release Note:
```
The federation-apiserver now supports a SchedulingPolicy admission controller that enables policy-based control over placement of federated resources.
```
Ref #39982
Automatic merge from submit-queue (batch tested with PRs 46967, 46992, 43338, 46717, 46672)
Add group alias names to API resources to allow discovery
**What this PR does / why we need it**:
Adds `GroupNames []string` to API resources, which represents the list of group aliases that every resource belongs to.
Partially fixes https://github.com/kubernetes/kubernetes/issues/41353
This moves the logic of "all" (which currently [translates](fbc94c0896/pkg/kubectl/cmd/util/shortcut_restmapper.go (L106)) to "pods,replicationcontrollers,services,...") to the server-side. Will allow clients like `kubectl` to discover group aliases instead of having it hardcoded and the API server to better handle consistency across multiple clients, version skew, etc; and will make "all" un-special and allow other groups to be created.
As a follow-up we'll patch `kubectl` to make groups aliases discoverable and the hardcoded list a fallback while we still have to support it.
Related to https://github.com/kubernetes/kubernetes/pull/42595#issuecomment-286839885.
**Release note**:
```release-note
Adds the `Categories []string` field to API resources, which represents the list of group aliases (e.g. "all") that every resource belongs to.
```
@kubernetes/sig-api-machinery-misc @deads2k @bgrant0607
Automatic merge from submit-queue (batch tested with PRs 46972, 42829, 46799, 46802, 46844)
promote tls-bootstrap to beta
last commit of this PR.
Towards https://github.com/kubernetes/kubernetes/issues/46999
```release-note
Promote kubelet tls bootstrap to beta. Add a non-experimental flag to use it and deprecate the old flag.
```
Automatic merge from submit-queue (batch tested with PRs 46550, 46663, 46816, 46820, 46460)
Add configuration for encryption providers
## Additions
Allows providing a configuration file (using flag `--experimental-encryption-provider-config`) to use the existing AEAD transformer (with multiple keys) by composing mutable transformer, prefix transformer (for parsing providerId), another prefix transformer (for parsing keyId), and AES-GCM transformers (one for each key). Multiple providers can be configured using the configuration file.
Example configuration:
```
kind: EncryptionConfig
apiVersion: v1
resources:
- resources:
- namespaces
providers:
- aes:
keys:
- name: key1
secret: c2vjcmv0iglzihnly3vyzq==
- name: key2
secret: dghpcybpcybwyxnzd29yza==
- identity: {}
```
Need for configuration discussed in:
#41939
[Encryption](3418b4e4c6/contributors/design-proposals/encryption.md)
**Pathway of a read/write request**:
1. MutableTransformer
2. PrefixTransformer reads the provider-id, and passes the request further if that matches.
3. PrefixTransformer reads the key-id, and passes the request further if that matches.
4. GCMTransformer tries decrypting and authenticating the cipher text in case of reads. Similarly for writes.
## Caveats
1. To keep the command line parameter parsing independent of the individual transformer's configuration, we need to convert the configuration to an `interface{}` and manually parse it in the transformer. Suggestions on better ways to do this are welcome.
2. Flags `--encryption-provider` and `--encrypt-resource` (both mentioned in [this document](3418b4e4c6/contributors/design-proposals/encryption.md) ) are not supported in this because they do not allow more than one provider, and the current format for the configuration file possibly supersedes their functionality.
3. Currently, it can be tested by adding `--experimental-encryption-provider-config=config.yml` to `hack/local-up-cluster.sh` on line 511, and placing the above configuration in `config.yml` in the root project directory.
Previous discussion on these changes:
https://github.com/sakshamsharma/kubernetes/pull/1
@jcbsmpsn @destijl @smarterclayton
## TODO
1. Investigate if we need to store keys on disk (per [encryption.md](3418b4e4c6/contributors/design-proposals/encryption.md (option-1-simple-list-of-keys-on-disk)))
2. Look at [alpha flag conventions](https://github.com/kubernetes/kubernetes/blob/master/pkg/features/kube_features.go)
3. Need to reserve `k8s:enc` prefix formally for encrypted data. Else find a better way to detect transformed data.
Automatic merge from submit-queue
Add initializer support to admission and uninitialized filtering to rest storage
Initializers are the opposite of finalizers - they allow API clients to react to object creation and populate fields prior to other clients seeing them.
High level description:
1. Add `metadata.initializers` field to all objects
2. By default, filter objects with > 0 initializers from LIST and WATCH to preserve legacy client behavior (known as partially-initialized objects)
3. Add an admission controller that populates .initializer values per type, and denies mutation of initializers except by certain privilege levels (you must have the `initialize` verb on a resource)
4. Allow partially-initialized objects to be viewed via LIST and WATCH for initializer types
5. When creating objects, the object is "held" by the server until the initializers list is empty
6. Allow some creators to bypass initialization (set initializers to `[]`), or to have the result returned immediately when the object is created.
The code here should be backwards compatible for all clients because they do not see partially initialized objects unless they GET the resource directly. The watch cache makes checking for partially initialized objects cheap. Some reflectors may need to change to ask for partially-initialized objects.
```release-note
Kubernetes resources, when the `Initializers` admission controller is enabled, can be initialized (defaulting or other additive functions) by other agents in the system prior to those resources being visible to other clients. An initialized resource is not visible to clients unless they request (for get, list, or watch) to see uninitialized resources with the `?includeUninitialized=true` query parameter. Once the initializers have completed the resource is then visible. Clients must have the the ability to perform the `initialize` action on a resource in order to modify it prior to initialization being completed.
```
Automatic merge from submit-queue (batch tested with PRs 46239, 46627, 46346, 46388, 46524)
Dynamic webhook admission control plugin
Unit tests pass.
Needs plumbing:
* [ ] service resolver (depends on @wfender PR)
* [x] client cert (depends on ????)
* [ ] hook source (depends on @caesarxuchao PR)
Also at least one thing will need to be renamed after Chao's PR merges.
```release-note
Allow remote admission controllers to be dynamically added and removed by administrators. External admission controllers make an HTTP POST containing details of the requested action which the service can approve or reject.
```
Automatic merge from submit-queue (batch tested with PRs 46801, 45184, 45930, 46192, 45563)
Exclude dockerized verify patterns
**What this PR does / why we need it**:
Change some of the `make verify` logic to allow excluding based on a pattern. Add the `verify-*-dockerized.sh` pattern to the excluded list.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 46801, 45184, 45930, 46192, 45563)
Tweak client-gen to support metrics API
This PR adds support to client-gen for readonly APIs as well as for customizing the resource name used for a given type.
This is required to support the clients generated for `k8s.io/metrics/pkg/apis/metrics`.
cc @caesarxuchao
Currently based on #45180
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 46432, 46701, 46326, 40848, 46396)
Add a server side Get operation
Implement proposal kubernetes/community#363
```release-note
The Kubernetes API supports retrieving tabular output for API resources via a new mime-type `application/json;as=Table;v=v1alpha1;g=meta.k8s.io`. The returned object (if the server supports it) will be of type `meta.k8s.io/v1alpha1` with `Table`, and contain column and row information related to the resource. Each row will contain information about the resource - by default it will be the object metadata, but callers can add the `?includeObject=Object` query parameter and receive the full object. In the future kubectl will use this to retrieve the results of `kubectl get`.
```
Add location transformer, config for transformers
Location transformer helps choose the most specific transformer for
read/write operations depending on the path of resource being accessed.
Configuration allows use of --experimental-encryption-provider-config
to set up encryption providers. Only AEAD is supported at the moment.
Add new files to BUILD, AEAD => k8s-aes-gcm
Use group resources to select encryption provider
Update tests for configuration parsing
Remove location transformer
Allow specifying providers per resource group in configuration
Add IdentityTransformer configuration option
Fix minor issues with initial AEAD implementation
Unified parsing of all configurations
Parse configuration using a union struct
Run configuration parsing in APIserver, refactor parsing
More gdoc, fix minor bugs
Add test coverage for combined transformers
Use table driven tests for encryptionconfig
Automatic merge from submit-queue
apiserver: add a webhook implementation of the audit backend
This builds off of #45315 and is intended to implement an interfaced defined in #45766.
TODO:
- [x] Rebase on top of API types PR.
- [x] Rebase on top of API types updates (#46065)
- [x] Rebase on top of feature flag (#46009)
- [x] Rebase on top of audit instrumentation.
- [x] Hook up API server flag or register plugin (depending on #45766)
Features issue https://github.com/kubernetes/features/issues/22
Design proposal https://github.com/kubernetes/community/blob/master/contributors/design-proposals/auditing.md
```release-notes
Webhook added to the API server which omits structured audit log events.
```
/cc @soltysh @timstclair @soltysh @deads2k
Automatic merge from submit-queue
Set Kubelet Disk Defaults for the 1.7 release
The `--low-diskspace-threshold-mb` flag has been depreciated since 1.6.
This PR sets the default to `0`, and sets defaults for disk eviction based on the values used for our [e2e tests](https://github.com/kubernetes/kubernetes/blob/master/test/e2e_node/services/kubelet.go#L145).
This also removes the custom defaults for vagrant, as the new defaults should work for it as well.
/assign @derekwaynecarr
cc @vishh
```release-note
By default, --low-diskspace-threshold-mb is not set, and --eviction-hard includes "nodefs.available<10%,nodefs.inodesFree<5%"
```
Automatic merge from submit-queue
Disable all alpha feature gates by default in local-up-cluster.sh
**What this PR does / why we need it**:
Disable all alpha feature gates by default in local-up-cluster.sh
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#46691
**Special notes for your reviewer**:
**Release note**:
```
None
```
Automatic merge from submit-queue
hack/verify-staging-imports.sh: simplify by negating package list
The forbidden list of repos got lengthy. The inverse is much shorter and easiert to maintain.
Automatic merge from submit-queue (batch tested with PRs 43505, 45168, 46439, 46677, 46623)
Add TPR to CRD migration helper.
This is a helper for migrating TPR data to CustomResource. It's rather hacky because it requires crossing apiserver boundaries, but doing it this way keeps the mess contained to the TPR code, which is scheduled for deletion anyway.
It's also not completely hands-free because making it resilient enough to be completely automated is too involved to be worth it for an alpha-to-beta migration, and would require investing significant effort to fix up soon-to-be-deleted TPR code. Instead, this feature will be documented as a best-effort helper whose results should be verified by hand.
The intended benefit of this over a totally manual process is that it should be possible to copy TPR data into a CRD without having to tear everything down in the middle. The process would look like this:
1. Upgrade to k8s 1.7. Nothing happens to your TPRs.
1. Create CRD with group/version and resource names that match the TPR. Still nothing happens to your TPRs, as the CRD is hidden by the overlapping TPR.
1. Delete the TPR. The TPR data is converted to CustomResource data, and the CRD begins serving at the same REST path.
Note that the old TPR data is left behind by this process, so watchers should not receive DELETE events. This also means the user can revert to the pre-migration state by recreating the TPR definition.
Ref. https://github.com/kubernetes/kubernetes/issues/45728
Automatic merge from submit-queue (batch tested with PRs 46686, 45049, 46323, 45708, 46487)
[Federation][kubefed]: Use StorageClassName for etcd pvc
This PR updates kubefed to use the StorageClassName field [added in 1.6](http://blog.kubernetes.io/2017/03/dynamic-provisioning-and-storage-classes-kubernetes.html
) for etcd's pvc to allow the user to specify which storage class they want to use. If no value is provided to ``kubefed init``, the field will not be set, and initialization of the pvc may fail on a cluster without a default storage class configured.
The alpha annotation that was previously used (``volume.alpha.kubernetes.io/storage-class``) was deprecated as of 1.4 according to the following blog post:
http://blog.kubernetes.io/2016/10/dynamic-provisioning-and-storage-in-kubernetes.html
**Release note**:
```
'kubefed init' has been updated to support specification of the storage class (via --etcd-pv-storage-class) for the Persistent Volume Claim (PVC) used for etcd storage. If --etcd-pv-storage-class is not specified, the default storage class configured for the cluster will be used.
```
cc: @kubernetes/sig-federation-pr-reviews
To properly register the types in the admission API group we need to
create an "install" package and wire it up. This is required by the
webhook admission controller being developed as part of
https://github.com/kubernetes/community/pull/132
Automatic merge from submit-queue
Node authorizer
This PR implements the authorization portion of https://github.com/kubernetes/community/blob/master/contributors/design-proposals/kubelet-authorizer.md and kubernetes/features#279:
* Adds a new authorization mode (`Node`) that authorizes requests from nodes based on a graph of related pods,secrets,configmaps,pvcs, and pvs:
* Watches pods, adds edges (secret -> pod, configmap -> pod, pvc -> pod, pod -> node)
* Watches pvs, adds edges (secret -> pv, pv -> pvc)
* When both Node and RBAC authorization modes are enabled, the default RBAC binding that grants the `system:node` role to the `system:nodes` group is not automatically created.
* Tightens the `NodeRestriction` admission plugin to require identifiable nodes for requests from users in the `system:nodes` group.
This authorization mode is intended to be used in combination with the `NodeRestriction` admission plugin, which limits the pods and nodes a node may modify. To enable in combination with RBAC authorization and the NodeRestriction admission plugin:
* start the API server with `--authorization-mode=Node,RBAC --admission-control=...,NodeRestriction,...`
* start kubelets with TLS boostrapping or with client credentials that place them in the `system:nodes` group with a username of `system:node:<nodeName>`
```release-note
kube-apiserver: a new authorization mode (`--authorization-mode=Node`) authorizes nodes to access secrets, configmaps, persistent volume claims and persistent volumes related to their pods.
* Nodes must use client credentials that place them in the `system:nodes` group with a username of `system:node:<nodeName>` in order to be authorized by the node authorizer (the credentials obtained by the kubelet via TLS bootstrapping satisfy these requirements)
* When used in combination with the `RBAC` authorization mode (`--authorization-mode=Node,RBAC`), the `system:node` role is no longer automatically granted to the `system:nodes` group.
```
```release-note
RBAC: the automatic binding of the `system:node` role to the `system:nodes` group is deprecated and will not be created in future releases. It is recommended that nodes be authorized using the new `Node` authorization mode instead. Installations that wish to continue giving all members of the `system:nodes` group the `system:node` role (which grants broad read access, including all secrets and configmaps) must create an installation-specific ClusterRoleBinding.
```
Follow-up:
- [ ] enable e2e CI environment with admission and authorizer enabled (blocked by kubelet TLS bootstrapping enablement in https://github.com/kubernetes/kubernetes/pull/40760)
- [ ] optionally enable this authorizer and admission plugin in kubeadm
- [ ] optionally enable this authorizer and admission plugin in kube-up
Automatic merge from submit-queue
Support grabbing test suite metrics
**What this PR does / why we need it**:
Add support for grabbing metrics that cover the entire test suite's execution.
Update the "interesting" controller-manager metrics to match the
current names for the garbage collector, and add namespace controller
metrics to the list.
If you enable `--gather-suite-metrics-at-teardown`, the metrics file is written to a file with a name such as `MetricsForE2ESuite_2017-05-25T20:25:57Z.json` in the `--report-dir`. If you don't specify `--report-dir`, the metrics are written to the test log output.
I'd like to enable this for some of the `pull-*` CI jobs, which will require a separate PR to test-infra.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
@kubernetes/sig-testing-pr-reviews @smarterclayton @wojtek-t @gmarek @derekwaynecarr @timothysc
This sample adds sample code for create/list/replace/delete operations for
Deployment resource for client-go library. This patch also contains
documentation explaining what the example application does and how it can be
executed.
This patch is part of the body of work improving the client library samples.
Signed-off-by: Ahmet Alp Balkan <ahmetb@google.com>
Automatic merge from submit-queue
Support storageclass storage updates to v1
**What this PR does / why we need it**: enable cluster administrators to update storageclasses stored in etcd from storage.k8s.io/v1beta1 to storage.k8s.io/v1.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Special notes for your reviewer**: I had a hard time getting the test to work with how it was handling KUBE_API_VERSIONS and RUNTIME_CONFIG. I would appreciate some extra review attention there. Also, I had to hack in a `cluster-scoped` "namespace" to get the verification portions of the test script to work. I'm definitely open to ideas for how to improve that if needed.
**Release note**:
```release-note
Support updating storageclasses in etcd to storage.k8s.io/v1. You must do this prior to upgrading to 1.8.
```
cc @kubernetes/sig-storage-pr-reviews @kubernetes/sig-api-machinery-pr-reviews @jsafrane @deads2k @saad-ali @enj
Automatic merge from submit-queue (batch tested with PRs 46561, 46566)
Fix golint errors in cmd/kube-apiserver
**What this PR does / why we need it**:
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
```
Automatic merge from submit-queue (batch tested with PRs 46561, 46566)
Return nil when err is nil
**What this PR does / why we need it**:
Return nil directly when err is nil
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue
Fix golint error in apimachinery/pkg/{fields,lables}
Also add these two packages to linted packages lists
**What this PR does / why we need it**:
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
```
Automatic merge from submit-queue
no-snat test
This test checks that Pods can communicate with each other in the same cluster without SNAT.
I intend to create a job that runs this in small clusters (\~3 nodes) at a low frequency (\~once per day) so that we have a signal as we work on allowing multiple non-masquerade CIDRs to be configured (see [kubernetes-incubator/ip-masq-agent](https://github.com/kubernetes-incubator/ip-masq-agent), for example).
/cc @dnardo
Automatic merge from submit-queue
Protobuf generation for k8s.io/metrics
This PR introduces protobuf generation for k8s.io/metrics. Doing so required:
- fixing a bug in `go-to-protobuf` causing the `cast{key,value,type}` values to not be quoted when coming from struct tags (and not auto-injection by `go-to-protobuf` itself).
- Making sure the proto IDL in k8s.io/client-go had a package name of `k8s.io.client_go.xyz` and not `k8s.io.kubernetes.xyz`.
Additionally, I updated `go-to-protobuf` to skip functions and non-public types when composing the import list, which cuts down on the more bizarre imports in the IDL (like importing the sample API package in every IDL file because it contained `addToScheme`, like every other API package).
We use `castvalue` to force gogo-proto to realize that it should consider the value of the map which underlies `ResourceList` when calculating which imports need to be named. Otherwise, it ignores the value's type, leading to compilation errors when it later can't find an import it assumed existed. We accidentally didn't hit this in `k8s.io/kubernetes/pkg/api/v1` since another field coincidentally happens to directly use `resource.Quantity` (the value type of `ResourceList`).
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 46302, 44597, 44742, 46554)
Change to aggregator so it calls a user apiservice via its pod IP.
proxy_handler now does a sideways call to lookup the pod IPs for aservice.
It will then pick a random pod IP to forward the use apiserver request to.
**What this PR does / why we need it**: It allows the aggregator to work without setting up the full network stack on the kube master (i.e. with kube-dns or kube-proxy)
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#44619
**Special notes for your reviewer**:
**Release note**:
```release-note
```
