Automatic merge from submit-queue (batch tested with PRs 63138, 63091, 63201, 63341). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Enable bypassing online checks in kubeadm upgrade plan
Signed-off-by: Chuck Ha <ha.chuck@gmail.com>
**What this PR does / why we need it**:
This PR makes `kubeadm upgrade plan` a little nicer to use in an air gapped environment. `kubeadm upgrade plan` now accepts a version and returns that instead of checking the internet.
**Which issue(s) this PR fixes**:
Fixeskubernetes/kubeadm#698
**Special notes for your reviewer**:
I also cleaned up the tests for this section of code by adding formal names for table tests and using `t.Run`.
**Release note**:
```release-note
`kubeadm upgrade plan` now accepts a version which improves the UX nicer in air-gapped environments.
```
Automatic merge from submit-queue (batch tested with PRs 63138, 63091, 63201, 63341). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Api server extra args
**What this PR does / why we need it**:
This PR will correctly override the APIServer command with the values found in APIServerExtraArgs. If none are passed in a default set of flags will be used.
This also includes cleaning up tests by using `t.Run` and naming test cases so we know what they are doing. `t.Run` also provides better testing output when coupled with a name.
At some point we stopped using the k8s version to generate this command. It is trivial to add back if we need it later, but since we don't use it now it's best for it to go. This also exposed several test that were identical. Those have been removed.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixeskubernetes/kubeadm#760
**Special notes for your reviewer**:
I split this into two commits since it's a bit easier to think about like that, but would be fine squashing.
**Release note**:
```release-note
NONE
```
/cc @kubernetes/sig-cluster-lifecycle-pr-reviews
`kubeadm upgrade plan <version>` is now supported. If no
version is supplied then the original behavior remains.
If a version is supplied there will be no pause when figuring out
versions. Kubeadm will assume the version you pass in is the latest
stable version.
Signed-off-by: Chuck Ha <ha.chuck@gmail.com>
Removes an unused parameter in getAPIServerCommand
Cleans up tests by:
* Naming the tests
* Using t.Run for better test output
* Removing duplicates
Fixeskubernets/kubeadm#760
Signed-off-by: Chuck Ha <ha.chuck@gmail.com>
Automatic merge from submit-queue (batch tested with PRs 59965, 59115, 63076, 63059). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Prepull etcd before an upgrade
If kubeadm ever has to upgrade etcd it should prepull the image so
there is less downtime during the upgrade when etcd versions change.
Fixeskubernetes/kubeadm#669
Signed-off-by: Chuck Ha <ha.chuck@gmail.com>
**What this PR does / why we need it**:
This PR Prepulls the etcd image during a `kubeadm upgrade apply`.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixeskubernetes/kubeadm#669
**Special notes for your reviewer**:
constants.MasterComponents was not changed because it is used in many places where etcd does not need to be nor should it be a part of this slice.
**Release note**:
```release-note
NONE
```
/cc @kubernetes/sig-cluster-lifecycle-pr-reviews
Automatic merge from submit-queue (batch tested with PRs 59965, 59115, 63076, 63059). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
kubeadm: prompt for confirmation when resetting a master
Signed-off-by: Alexander Brand <alexbrand09@gmail.com>
**What this PR does / why we need it**:
This PR implements a confirmation prompt when running `kubeadm reset` on a master node. This is to prevent users from mistakenly resetting a master node.
**Which issue(s) this PR fixes**:
Fixes https://github.com/kubernetes/kubeadm/issues/673
**Special notes for your reviewer**:
I am somewhat torn on the approach on how to detect that kubeadm is running on a master node. I went with checking for the apiserver manfiest file on the local filesystem, as it seems like a simpler approach when compared to getting a k8s client, getting a list of nodes, finding the current node, and checking if it has the master taint. I am happy to rework if the latter is more desirable.
Sample runs:
```
# ./kubeadm reset
[Warning] Are you sure you want to reset this master node? Type the word "confirm" to continue: no
Aborted reset operation on master node
# ./kubeadm reset
[Warning] Are you sure you want to reset this master node? Type the word "confirm" to continue: confirm
[preflight] Running pre-flight checks.
[reset] Stopping the kubelet service.
[reset] WARNING: The kubelet service could not be stopped by kubeadm: [exit status 1]
[reset] WARNING: Please ensure kubelet is stopped manually.
[reset] Unmounting mounted directories in "/var/lib/kubelet"
........
# ./kubeadm reset
[Warning] Are you sure you want to reset this master node? Type the word "confirm" to continue:
Aborted reset operation on master node
# ./kubeadm reset --confirm
[preflight] Running pre-flight checks.
[reset] Stopping the kubelet service.
[reset] WARNING: The kubelet service could not be stopped by kubeadm: [exit status 1]
[reset] WARNING: Please ensure kubelet is stopped manually.
[reset] Unmounting mounted directories in "/var/lib/kubelet"
........
```
**Release note**:
```release-note
kubeadm: prompt the user for confirmation when resetting a master node
```
Automatic merge from submit-queue (batch tested with PRs 59965, 59115, 63076, 63059). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
fix help message of kubeconfig-dir option(kubeadm alpha phase kubeconfig all)
**What this PR does / why we need it**:
This patch will fix wrong help message.
The command is kubeadm alpha phase kubeconfig [command]
The help message is for --kubeconfig-dir option.
kubeconfig-dir is not port.(It is directory)
So, I fixed the message.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Special notes for your reviewer**:
**Release note**:
`NONE`
/sig cluster-lifecycle
Automatic merge from submit-queue (batch tested with PRs 61601, 62881, 63159). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
support simultaneous kubeadm --v and --config
**What this PR does / why we need it**:
Providing -v and --config parameters to increase verbosity while providing a kubeadm.config results in an error rather than providing the requested verbosity.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes https://github.com/kubernetes/kubeadm/issues/765
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 61601, 62881, 63159). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
kubeadm: add test coverage to join.go
**What this PR does / why we need it**:
Add test coverage to `join.go`.
A separate commit exports the function `ValidateJoinCommandLine()` from `join.go` so that testing this file is more flexible.
Test coverage is at 76%. One untested part is successfully running `Join.Run()` without errors, but that requires a valid HTTPS API server running and a valid config. i got this partially working but gave up because i faced some cert / config blockers. suggestions on how to get that to work easily are welcome.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
NONE
**Special notes for your reviewer**:
NONE
**Release note**:
```release-note
NONE
```
Instead of using kubeadmutil.CheckErr() in every single
phase of cmd.Run(), use a new helper function
NewValidJoin() that returns a single error.
This would improve the unit testing options for this file.
Otherwise any error in cmd.Run() will trigger an os.Exit()
as kubeadmutil.CheckErr() does that.
Automatic merge from submit-queue (batch tested with PRs 59220, 62927, 63084, 63090, 62284). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
kubeadm: add better test coverage to token.go
**What this PR does / why we need it**:
a PR for adding some more tests in `kubeadm/cmd` for `token.go`.
some areas of the `token.go` like listing, creating and deleting tokens can present challenges.
coverage was increased to around 87%.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
please, link issue # if you know of such.
**Special notes for your reviewer**:
none
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 62655, 61711, 59122, 62853, 62390). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
kubeadm: Make kube-proxy tolerate all taints
**What this PR does / why we need it**:
As a essential core component, kube-proxy should generally run on all
nodes even if the cluster operator taints nodes for special purposes.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixeskubernetes/kubeadm#699
**Release note**:
```release-note
kubeadm creates kube-proxy with a toleration to run on all nodes, no matter the taint.
```
Automatic merge from submit-queue (batch tested with PRs 62655, 61711, 59122, 62853, 62390). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
kubeadm: Mount additional paths inside apiserver/controller-manager for working CA root
This is required for a working CA root, as /etc/ssl/certs on a few
Linux distributions just contains a bunch of symlinks.
Container Linux and Debian have symlinks pointing to
/usr/share/ca-certificates, ArchLinux has symlinks pointing
to /etc/ca-certificates.
On Debian /etc/ssl/certs can also include symlinks pointing
to /usr/local/share/ca-certificates for local CA certificates.
Fix: kubeadm/#671
---
**What this PR does / why we need it**:
Without this PR, `controller-manager` and `apiserver` would lack a CA root on some Linux distro (ex: Container Linux) which for example break flexplugins which require a CA root [[1]](https://github.com/kubernetes-incubator/external-storage/issues/571#issuecomment-360155462).
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes https://github.com/kubernetes/kubeadm/issues/671
**Special notes for your reviewer**:
**Release note**:
```release-note
Mount additional paths required for a working CA root, for setups where /etc/ssl/certs doesn't contains certificates but just symlink.
```
/sig sig-kubeadm
Automatic merge from submit-queue (batch tested with PRs 62655, 61711, 59122, 62853, 62390). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Modify the kubeadm upgrade DAG for the TLS Upgrade
**What this PR does / why we need it**:
This adds the necessary utilities to detect Etcd TLS on static pods from the file system and query Etcd.
It modifies the upgrade logic to make it support the APIServer downtime.
Tests are included and should be passing.
```bash
bazel test //cmd/kubeadm/... \
&& bazel build //cmd/kubeadm --platforms=@io_bazel_rules_go//go/toolchain:linux_amd64 \
&& issue=TLSUpgrade ~/Repos/vagrant-kubeadm-testing/copy_kubeadm_bin.sh
```
These cases are working consistently for me
```bash
kubeadm-1.9.6 reset \
&& kubeadm-1.9.6 init --kubernetes-version 1.9.1 \
&& kubectl apply -f https://git.io/weave-kube-1.6
/vagrant/bin/TLSUpgrade_kubeadm upgrade apply 1.9.6 # non-TLS to TLS
/vagrant/bin/TLSUpgrade_kubeadm upgrade apply 1.10.0 # TLS to TLS
/vagrant/bin/TLSUpgrade_kubeadm upgrade apply 1.10.1 # TLS to TLS
/vagrant/bin/TLSUpgrade_kubeadm upgrade apply 1.9.1 # TLS to TLS /w major version downgrade
```
This branch is based on top of #61942, as resolving the hash race condition is necessary for consistent behavior.
It looks to fit in pretty well with @craigtracey's PR: #62141
The interfaces are pretty similar
/assign @detiber @timothysc
**Which issue(s) this PR fixes**
Helps with https://github.com/kubernetes/kubeadm/issues/740
**Special notes for your reviewer**:
278b322a1c
[kubeadm] Implement ReadStaticPodFromDisk
c74b56372d
Implement etcdutils with Cluster.HasTLS()
- Test HasTLS()
- Instrument throughout upgrade plan and apply
- Update plan_test and apply_test to use new fake Cluster interfaces
- Add descriptions to upgrade range test
- Support KubernetesDir and EtcdDataDir in upgrade tests
- Cover etcdUpgrade in upgrade tests
- Cover upcoming TLSUpgrade in upgrade tests
8d8e5fe33b
Update test-case, fix nil-pointer bug, and improve error message
97117fa873
Modify the kubeadm upgrade DAG for the TLS Upgrade
- Calculate `beforePodHashMap` before the etcd upgrade in anticipation of
KubeAPIServer downtime
- Detect if pre-upgrade etcd static pod cluster `HasTLS()==false` to switch
on the Etcd TLS Upgrade if TLS Upgrade:
- Skip L7 Etcd check (could implement a waiter for this)
- Skip data rollback on etcd upgrade failure due to lack of L7 check
(APIServer is already down unable to serve new requests)
- On APIServer upgrade failure, also rollback the etcd manifest to
maintain protocol compatibility
- Add logging
**Release note**:
```release-note
kubeadm upgrade no longer races leading to unexpected upgrade behavior on pod restarts
kubeadm upgrade now successfully upgrades etcd and the controlplane to use TLS
kubeadm upgrade now supports external etcd setups
kubeadm upgrade can now rollback and restore etcd after an upgrade failure
```
Fix `rollbackEtcdData()` to return error=nil on success
`rollbackEtcdData()` used to always return an error making the rest of the
upgrade code completely unreachable.
Ignore errors from `rollbackOldManifests()` during the rollback since it
always returns an error.
Success of the rollback is gated with etcd L7 healthchecks.
Remove logic implying the etcd manifest should be rolled back when
`upgradeComponent()` fails
- Calculate `beforePodHashMap` before the etcd upgrade in anticipation of KubeAPIServer downtime
- Detect if pre-upgrade etcd static pod cluster `HasTLS()==false` to switch on the Etcd TLS Upgrade
if TLS Upgrade:
- Skip L7 Etcd check (could implement a waiter for this)
- Skip data rollback on etcd upgrade failure due to lack of L7 check (APIServer is already down unable to serve new requests)
- On APIServer upgrade failure, also rollback the etcd manifest to maintain protocol compatibility
- Add logging
- Test HasTLS()
- Instrument throughout upgrade plan and apply
- Update plan_test and apply_test to use new fake Cluster interfaces
- Add descriptions to upgrade range test
- Support KubernetesDir and EtcdDataDir in upgrade tests
- Cover etcdUpgrade in upgrade tests
- Cover upcoming TLSUpgrade in upgrade tests
If kubeadm ever has to upgrade etcd it should prepull the image so
there is less downtime during the upgrade when etcd versions change.
Fixeskubernetes/kubeadm#669
Signed-off-by: Chuck Ha <ha.chuck@gmail.com>
Automatic merge from submit-queue (batch tested with PRs 62464, 62947). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
make API.ControlPlaneEndpoint accept IP
**What this PR does / why we need it**:
This PR implements one of the actions defined by https://github.com/kubernetes/kubeadm/issues/751 (checklist form implementing HA in kubeadm).
With this PR, the `API.ControlPlaneEndpoint` value in the kubeadm MasterConfiguration file now accepts both DNS and IP.
The `API.ControlPlaneEndpoint` should be used to set a stable IP address for the control plane; in an HA configuration, this should be the load balancer address (no matter if identified by a DNS name or by a stable IP).
**Special notes for your reviewer**:
/CC @timothysc
This PR is the same of https://github.com/kubernetes/kubernetes/pull/62667, that I closed by error 😥
**Release note**:
```release-note
NONE
```
Nb. first https://github.com/kubernetes/kubernetes/pull/62667 already has the release note
- Update kubeadm static pod upgrades to use the
kubetypes.ConfigHashAnnotationKey annotation on the mirror pod rather
than generating a hash from the full object info. Previously, a status
update for the pod would allow the upgrade to proceed before the
new static pod manifest was actually deployed.
Signed-off-by: Jason DeTiberus <detiber@gmail.com>
Automatic merge from submit-queue (batch tested with PRs 62726, 60085, 62583). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Refactor kubeadm api validation.
**What this PR does / why we need it**:
This PR refactor kubeadm api validation to use field json name. When users get a validation error, they can easily know which field is invalid in their config files.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
@fabriziopandini I remember we mentioned this refactoring in some comment. I didn't see this change was done, so I send this PR to address this.
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 62781, 62815). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Update kubeadm upgrade message
**What this PR does / why we need it**:
Updates kubeadm upgrade message per user request
**Which issue(s) this PR fixes**:
Fixes https://github.com/kubernetes/kubeadm/issues/672
**Special notes for your reviewer**:
none
**Release note**:
```
NONE
```
/cc @kubernetes/sig-cluster-lifecycle-pr-reviews @kargakis
Automatic merge from submit-queue (batch tested with PRs 62481, 62643, 61877, 62515). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Not validating front proxy CA Key when using External CA.
**What this PR does / why we need it**:
"That the front ca key is not required as the front proxy client tls keypair can be managed by the third party." This PR don't validate the front CA Key but check if it already exists.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes https://github.com/kubernetes/kubeadm/issues/752
**Special notes for your reviewer**:
@yanndegat @timothysc @stealthybox @fabriziopandini
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 62481, 62643, 61877, 62515). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
kubeadm preflight: check CRI socket path if defined or docker service otherwise
**What this PR does / why we need it**:
Currently, running `kubeadm init` without Docker installed will cause the `Service-Docker` preflight check to fail even when another CRI is installed and the CRI socket specified. This changes the preflight checks to check the CRI socket if specified, and falling back to checking the Docker service otherwise. Additionally, this deduplicates common checks between `kubeadm init` and `kubeadm join` to ensure that similar preflight checks stay in-sync going forward.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes # https://github.com/kubernetes/kubeadm/issues/657 - it still has the same behavior on joins
Fixes # https://github.com/kubernetes/kubeadm/issues/749 - will check the CRI socket if specified and skip the Docker service check
**Special notes for your reviewer**:
**Release note**:
```release-note
kubeadm preflight: check CRI socket path if defined, otherwise check for Docker
```
This is required for a working CA root, as /etc/ssl/certs on a few
Linux distributions just contains a bunch of symlinks.
Container Linux and Debian have symlinks pointing to
/usr/share/ca-certificates, ArchLinux has symlinks pointing
to /etc/ca-certificates.
On Debian /etc/ssl/certs can also include symlinks pointing
to /usr/local/share/ca-certificates for local CA certificates.
Fix: kubeadm/#671
Automatic merge from submit-queue (batch tested with PRs 62568, 62220, 62743, 62751, 62753). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Kubeadm upgrade same version
What this PR does / why we need it:
When kubeadm 1.10 came out, it inadvertently introduced a backwards incompatible config change. Because the kubeadm MasterConfiguration is written by the old version of kubeadm and read by the new one, this incompatibility causes the upgrade to fail.
To mitigate this, I've written a simple transform that operates on a map-based version of the config. This map is mutated to make it compatible with the new structure, then serialised to JSON and deserialised by the usual APIMachinery.
Because of complications with the multiple versions, this PR enforces kubeadm only being used to upgrade to kubernetes of the same minor and major versions.
Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes [kubeadm#744](https://github.com/kubernetes/kubeadm/issues/744#issuecomment-379045823L)
This PR is an alternate take on #62353. Instead of trying to gate migration on versions, this constrains kubeadm to only upgrade versions from the same major and minor versions.
Special notes for your reviewer:
```release-note
fixes configuration error when upgrading kubeadm from 1.9 to 1.10+
enforces kubeadm upgrading kubernetes from the same major and minor versions as the kubeadm binary.
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add phase command for dynamic kubelet configuration in kubeadm.
**What this PR does / why we need it**:
As the title says.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
partially fixes this: https://github.com/kubernetes/kubeadm/issues/571
**Special notes for your reviewer**:
/cc @luxas @fabriziopandini
@kubernetes/sig-cluster-lifecycle-pr-reviews
**Release note**:
```release-note
Phase `kubeadm alpha phase kubelet` is added to support dynamic kubelet configuration in kubeadm.
```
