Automatic merge from submit-queue (batch tested with PRs 50749, 52869, 53359). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Kubeadm: Change the marshal code to use ApiMachinery code.
**What this PR does / why we need it**:
The PR change the k8s obj marshaling to use ApiMachinery code instead of plain yaml.Marshal which is known to have some side-effect.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
fixeskubernetes/kubeadm#453
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
As of 1.8.0 kubelet in kubeadm configuration ships with enabled feature
of node certificate autorotation, it makes sense to enable automatic
certificate rotation csr signing. It will help to avoid issues like
described in #53231 and #53237.
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Allow to use version labels in kubeadm upgrade apply command.
**What this PR does / why we need it**:
kubeadm upgrade apply now is able to utilize all possible combinations
of version argument, including labels (latest, stable-1.8, ci/latest-1.9)
as well as specific builds (v1.8.0-rc.1, ci/v1.9.0-alpha.1.123_01234567889)
As side effect, specifying exact build to deploy from CI area is now also
possible in kubeadm init command.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixeskubernetes/kubeadm#451
**Special notes for your reviewer**:
cc @luxas
**Release note**:
```release-note
- kubeadm init can now deploy exact build from CI area by specifying ID with "ci/" prefix. Example: "ci/v1.9.0-alpha.1.123+01234567889"
- kubeadm upgrade apply supports all standard ways of specifying version via labels. Examples: stable-1.8, latest-1.8, ci/latest-1.9 and similar.
```
kubeadm upgrade apply now is able to utilize all possible combinations
of version argument, including labels (latest, stable-1.8, ci/latest-1.9)
as well as specific builds (v1.8.0-rc.1, ci/v1.9.0-alpha.1.123_01234567889)
As side effect, specifying exact build to deploy from CI area is now also
possible in kubeadm init command.
Fixes: kubernetes/kubeadm#451
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>..
bazel: build/test almost everything
**What this PR does / why we need it**: Miscellaneous cleanups and bug fixes. The main motivating idea here was to make `bazel build //...` and `bazel test //...` mostly work. (There's a few reasons these still don't work, but we're a lot closer.)
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
/assign @BenTheElder @mikedanese @spxtr
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>..
Add more test coverage for kubeadm uploadconfig especially with idemp…
**What this PR does / why we need it**:
This PR adds more test case for Kubeadm uploadconfig, particularly to address some feedback in #51482
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
fixes #kubernetes/kubeadm#379
**Special notes for your reviewer**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>..
Modify `apimachinery` imports using `staging`
**What this PR does / why we need it**:
Currently some of the imports of `apimachinery` use
`k8s.io/kubernetes/staging/src/k8s.io/apimachinery...`. Replace
these with `k8s.io/apimachinery`, as is in use throughout the rest
of the code base.
Signed-off-by: mattjmcnaughton <mattjmcnaughton@gmail.com>
**Release note**:
```release-note
NONE
```
Currently some of the imports of `apimachinery` use
`k8s.io/kubernetes/staging/src/k8s.io/apimachinery...`. Replace
these with `k8s.io/apimachinery`, as is in use throughout the rest
of the code base.
Signed-off-by: mattjmcnaughton <mattjmcnaughton@gmail.com>
Automatic merge from submit-queue
kubeadm: add `kubeadm phase addons` command
**What this PR does / why we need it**:
Adds the `addons` phase command to `kubeadm`
fixes: https://github.com/kubernetes/kubeadm/issues/418
/cc @luxas
Automatic merge from submit-queue (batch tested with PRs 51682, 51546, 51369, 50924, 51827)
kubeadm: Detect kubelet readiness and error out if the kubelet is unhealthy
**What this PR does / why we need it**:
In order to improve the UX when the kubelet is unhealthy or stopped, or whatever, kubeadm now polls the kubelet's API after 40 and 60 seconds, and then performs an exponential backoff for a total of 155 seconds.
If the kubelet endpoint is not returning `ok` by then, kubeadm gives up and exits.
This will miligate at least 60% of our "[apiclient] Created API client, waiting for control plane to come up" issues in the kubeadm issue tracker 🎉, as kubeadm now informs the user what's wrong and also doesn't deadlock like before.
Demo:
```
lucas@THEGOPHER:~/luxas/kubernetes$ sudo ./kubeadm init --skip-preflight-checks
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[init] Using Kubernetes version: v1.7.4
[init] Using Authorization modes: [Node RBAC]
[preflight] Skipping pre-flight checks
[kubeadm] WARNING: starting in 1.8, tokens expire after 24 hours by default (if you require a non-expiring token use --token-ttl 0)
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [thegopher kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.1.115]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "scheduler.conf"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests"
[init] This often takes around a minute; or longer if the control plane images have to be pulled.
[apiclient] All control plane components are healthy after 40.502199 seconds
[markmaster] Will mark node thegopher as master by adding a label and a taint
[markmaster] Master thegopher tainted and labelled with key/value: node-role.kubernetes.io/master=""
[bootstraptoken] Using token: 5776d5.91e7ed14f9e274df
[bootstraptoken] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[uploadconfig] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[addons] Applied essential addon: kube-dns
[addons] Applied essential addon: kube-proxy
Your Kubernetes master has initialized successfully!
To start using your cluster, you need to run (as a regular user):
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
http://kubernetes.io/docs/admin/addons/
You can now join any number of machines by running the following on each node
as root:
kubeadm join --token 5776d5.91e7ed14f9e274df 192.168.1.115:6443 --discovery-token-ca-cert-hash sha256:6f301ce8c3f5f6558090b2c3599d26d6fc94ffa3c3565ffac952f4f0c7a9b2a9
lucas@THEGOPHER:~/luxas/kubernetes$ sudo ./kubeadm reset
[preflight] Running pre-flight checks
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Removing kubernetes-managed containers
[reset] Deleting contents of stateful directories: [/var/lib/kubelet /etc/cni/net.d /var/lib/dockershim /var/run/kubernetes /var/lib/etcd]
[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
lucas@THEGOPHER:~/luxas/kubernetes$ sudo systemctl stop kubelet
lucas@THEGOPHER:~/luxas/kubernetes$ sudo ./kubeadm init --skip-preflight-checks
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[init] Using Kubernetes version: v1.7.4
[init] Using Authorization modes: [Node RBAC]
[preflight] Skipping pre-flight checks
[kubeadm] WARNING: starting in 1.8, tokens expire after 24 hours by default (if you require a non-expiring token use --token-ttl 0)
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [thegopher kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.1.115]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "scheduler.conf"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests"
[init] This often takes around a minute; or longer if the control plane images have to be pulled.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp 127.0.0.1:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp 127.0.0.1:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp 127.0.0.1:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp 127.0.0.1:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp 127.0.0.1:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp 127.0.0.1:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp 127.0.0.1:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp 127.0.0.1:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp 127.0.0.1:10255: getsockopt: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by that:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
- There is no internet connection; so the kubelet can't pull the following control plane images:
- gcr.io/google_containers/kube-apiserver-amd64:v1.7.4
- gcr.io/google_containers/kube-controller-manager-amd64:v1.7.4
- gcr.io/google_containers/kube-scheduler-amd64:v1.7.4
You can troubleshoot this for example with the following commands if you're on a systemd-powered system:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
couldn't initialize a Kubernetes cluster
```
In this demo, I'm first starting kubeadm normally and everything works as usual.
In the second case, I'm explicitely stopping the kubelet so it doesn't run, and skipping preflight checks, so that kubeadm doesn't even try to exec `systemctl start kubelet` like it does usually.
That obviously results in a non-working system, but now kubeadm tells the user what's the problem instead of waiting forever.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
Fixes: https://github.com/kubernetes/kubeadm/issues/377
**Special notes for your reviewer**:
**Release note**:
```release-note
kubeadm: Detect kubelet readiness and error out if the kubelet is unhealthy
```
@kubernetes/sig-cluster-lifecycle-pr-reviews @pipejakob
cc @justinsb @kris-nova @lukemarsden as well as you wanted this feature :)
Automatic merge from submit-queue (batch tested with PRs 50832, 51119, 51636, 48921, 51712)
kubeadm: Add support for using an external CA whose key is never stored in the cluster
We allow a kubeadm user to use an external CA by checking to see if ca.key is missing and skipping cert checks and kubeconfig generation if ca.key is missing. We also pass an empty arg --cluster-signing-key-file="" to kube controller manager so that the csr signer doesn't start.
**What this PR does / why we need it**:
This PR allows the kubeadm certs phase and kubeconfig phase to be skipped if the ca.key is missing but all other certs are present.
**Which issue this PR fixes** :
Fixes kubernetes/kubeadm/issues/280
**Special notes for your reviewer**:
@luxas @mikedanese @fabriziopandini
**Release note**:
```release-note
kubeadm: Add support for using an external CA whose key is never stored in the cluster
```
Automatic merge from submit-queue
kubeadm: Add node-cidr-mask-size to pass to kube-controller-manager for IPv6
Due to the increased size of subnets with IPv6, the node-cidr-mask-size needs to be passed to kube-controller-manager. If IPv4 it will be set to 24 as it was previously, if IPv6, it will be set to
64
**What this PR does / why we need it**:
If the user specifies the --pod-network-cidr with kubeadm init, this caused the kube-controller-manager manifest to include the "--allocate-node-cidrs" and "--cluster-cidr" flags to be set. The --node-cidr-mask-size is not set, and currently defaults to 24, which is fine for IPv4, but not appropriate for IPv6. This change passes the a value as the node-cidr-mask-size to the controller-manager. It detects if it is IPv4 or v6, and sets --node-cidr-mask-size to 24 for IPv4 as before, and to 64 for IPv6.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#50469
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
We allow a kubeadm user to use an external CA by checking to see if ca.key is missing and skipping cert checks and kubeconfig generation if ca.key is missing.
Automatic merge from submit-queue (batch tested with PRs 49861, 50933, 51380, 50688, 51305)
Add configurable groups to bootstrap tokens.
**What this PR does / why we need it**:
This change adds support for authenticating bootstrap tokens into a configurable set of extra groups in addition to `system:bootstrappers`. Previously, bootstrap tokens could only ever authenticate to the `system:bootstrappers` group.
Groups are specified as a comma-separated list in the `auth-extra-groups` key of the `bootstrap.kubernetes.io/token` Secret, and must begin with the prefix `system:bootstrapper:` (and match a validation regex that checks against our normal convention). Whether or not any extra groups are configured, `system:bootstrappers` will still be added.
This also adds a `--groups` flag for `kubeadm token create`, which sets the `auth-extra-groups` key on the resulting Secret. The default is to not set the key.
`kubeadm token list` is also updated to include a `EXTRA GROUPS` output column.
**Which issue this PR fixes**: fixes#49306
**Special notes for your reviewer**:
The use case for this is in https://github.com/kubernetes/kubernetes/issues/49306. Comments on the feature itself are probably better over there. It will be part of how HA/self-hosting kubeadm bootstraps new master nodes (post 1.8).
**Release note**:
```release-note
Add support for configurable groups for bootstrap token authentication.
```
cc @luxas @kubernetes/sig-cluster-lifecycle-api-reviews @kubernetes/sig-auth-api-reviews
/kind feature
Automatic merge from submit-queue
kubeadm: Rename FeatureFlags to FeatureGates
**What this PR does / why we need it**:
Automatic rename from `FeatureFlags` to `FeatureGates`, as I noticed that's the real name for this feature. This is for consistency in the API and generally in the code.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
@kubernetes/sig-cluster-lifecycle-pr-reviews @fabriziopandini @jamiehannaford
Automatic merge from submit-queue (batch tested with PRs 51054, 51101, 50031, 51296, 51173)
Add host mountpath to controller-manager for flexvolume dir
Controller manager needs access to Flexvolume plugin when using attach-detach controller interface.
This PR adds the host mount path for the default directory of flexvolume plugins
Fixes https://github.com/kubernetes/kubeadm/issues/410
Controller manager needs access to Flexvolume plugin when
using attach-detach controller interface.
This PR adds the host mount path for the default directory of flexvolume
plugins
Fixes https://github.com/kubernetes/kubeadm/issues/410
Automatic merge from submit-queue (batch tested with PRs 51038, 50063, 51257, 47171, 51143)
update related manifest files to use hostpath type
**What this PR does / why we need it**:
Per [discussion in #46597](https://github.com/kubernetes/kubernetes/pull/46597#pullrequestreview-53568947)
Dependes on #46597
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
Fixes: https://github.com/kubernetes/kubeadm/issues/298
**Special notes for your reviewer**:
/cc @euank @thockin @tallclair @Random-Liu
**Release note**:
```release-note
None
```
Automatic merge from submit-queue (batch tested with PRs 51039, 50512, 50546, 50965, 50467)
kubeadm: Get kube-dns based on the kubernetes version
**What this PR does / why we need it**:
Makes the kube-dns version used dependent on the kubernetes version. This is required for upgrades as we have to be able to handle one kube-dns version per branch for instance...
Currently a no-op though, as both v1.7 and v1.8 seem to use 1.14.4
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Special notes for your reviewer**:
Dependency for https://github.com/kubernetes/kubernetes/pull/48899 (kubeadm upgrades)
**Release note**:
```release-note
NONE
```
@kubernetes/sig-cluster-lifecycle-pr-reviews
@kubernetes/dns-maintainers FYI; next time you bump DNS version, please update this func instead of the constant there...
