When using hack/local-up-cluster.sh deploy local cluster, it
failed with following message "kube-proxy terminated unexpectedly"
and "Failed to retrieve node info: nodes "127.0.0.1" not found" in
kube-proxy.log.
The root reason for this error is miss boot order of kubernetes
services in local-up-cluster.sh, kube-proxy and kubectl daemon.
When starting kube-proxy, it would check node information. And
these information are collected by kubelet daemon. However, in
the shell script, kube-proxy service start before kubelet daemon.
This patch changed the boot order of kubelet daemon and kube-proxy
and check if node stats ready for kube-proxy start.
Signed-off-by: Howard Zhang <howard.zhang@arm.com>
We should just use the regular binaries used everywhere else and drop
hyperkube. Only the local-up-cluster related CI jobs use hyperkube at
the moment, so this will help deprecate and eliminate hyperkube at least
from the main kubernetes repository.
Change-Id: Ic53a4b56420565f546ac75be6cc23d214766c057
The changes to local-cluster-up.sh from #75662 is causing the
CRI-O CI to break since it picks up the --runtime-request-timeout
flag as part of the container-runtime-endpoint flag.
Adding a new env so that we can set the runtime-request-timeout
value for our CI.
Signed-off-by: Urvashi Mohnani <umohnani@redhat.com>
Some CNIs such as OVN don't use kubeproxy and provide
there own implementation for network proxy .
This patch allow to disable the kubeproxy start in the
local-up-cluster.sh script.
It is also move the generation of the certificate and
kubeconfig to start_kubeproxy function so it will be
generated only when it needed.
This is useful for us for developement and CI testing.
Signed-off-by: Moshe Levi <moshele@mellanox.com>
Currently the kubescheduler starts in the start_kubeproxy
This change move it to ir own function start_kubescheduler
Change-Id: Iff93114d4becabe4b6b937c5077821e092abffd3
Signed-off-by: Moshe Levi <moshele@mellanox.com>
Running local-up-cluster with START_MODE='kubeletonly'
requires cert and kubeconfig. Currently, all the
certs and the kubeconfig are created in the start_apiserver
so it won't be created when only kubelet is started.
This change moves the creation of the kubelet cert and
kubeconfig to start_kubelet. This is helpful if you
want to run kubelet with local-up-cluster and to connect
to an existing master node.
Change-Id: Iaba0bf975297346fbede44342f163024caa256cf
Signed-off-by: Moshe Levi <moshele@mellanox.com>
recent changes to kubekins broke local-e2e with the following error
```
E: Unable to locate package sudo
```
Change-Id: I9ad324a2a070bc068ed1f0f88a912eafb191ad90
This change includes the yaml files and gce startup script changes
to run this addon. It is disabled by default, can be enabled by setting
KUBE_ENABLE_NODELOCAL_DNS=true
An ip address is required for the cache instance to listen for
requests on, default is a link local ip address of value 169.254.25.10
addressed review comments, updated image location
Picked a different prometheus port so stats port is not same as the
coredns deployment
Removed the nodelocaldns-ready label.
Set memory limit to 30Mi
Automatic merge from submit-queue (batch tested with PRs 66489, 66728, 66739). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
remove incomplete check of ipvs modules in hack/local-up-cluster.sh
**What this PR does / why we need it**:
Currently `hack/local-up-cluster.sh` executes `sudo modprobe -a ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack_ipv4` to check whether the ipvs required modules exist, which leaves out the scenario https://github.com/kubernetes/kubernetes/issues/63801 mentioned.
Since `func CanUseIPVSProxier` in `pkg/proxy/ipvs/proxier.go` covers all scenarios, maybe we should just remove this part instead of adding codes.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Switch off leader election for scheduler and kube/cloud controller
**What this PR does / why we need it**:
We now have leader election on by default, for a single node
local-up-cluster, this is not needed. Let's switch it off
This will reduce the flakiness and timeouts we see in the local e2e CI jobs.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
enable etcd logging in local-e2e jobs
**What this PR does / why we need it**:
We are not saving the etcd logs and just redirecting the output
to /dev/null. In this change, we set ETCD_LOGFILE to the same
directory where we log other kube relates processes.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
See example run in http://gcsweb.k8s.io/gcs/kubernetes-jenkins/pr-logs/pull/66608/pull-kubernetes-local-e2e/254/artifacts/kubetest-local263115757/
**Release note**:
```release-note
NONE
```
We now have leader election on by default, for a single node
local-up-cluster, this is not needed. Let's switch it off
Change-Id: I5cc4c3b9f74e318f378f239d1e8ee84603b3a6ef
We are not saving the etcd logs and just redirecting the output
to /dev/null. In this change, we set ETCD_LOGFILE to the same
directory where we log other kube relates processes.
Change-Id: I719a941665b749b864f2efdcdfc7ddc7b2a61776
Automatic merge from submit-queue (batch tested with PRs 60150, 65467, 65487, 65595, 65374). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add warning function that includes timestamp
Knowing when a process has terminated is helpful when looking through log files. The scenario that happened to me was that /tmp filled up, so as expected things failed. But I'd really like to know when things got bad without looking through 10G log files.
Release note:
```release-note
NONE
```
Knowing when a process has terminated is helpful when looking through
log files (especially very large ones). This also refactors the terminal
coloring function to be used with custom prefix and colors.
warning_log outputs red text in the current e2e inspired format:
W0628 16:58:26]: this is a test
Automatic merge from submit-queue (batch tested with PRs 61610, 64591, 58143, 63929). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Create CoreDNS and kube-dns folders
**What this PR does / why we need it**:
Separate the CoreDNS and kube-dns manifests by creating their own folders (dns/coredns and dns/kube-dns)
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#61435
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
cc @MrHohn
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add optional flag of node port range
**What this PR does / why we need it**:
Add option of setting port range when using hack/local-up-cluster.sh
**Which issue(s) this PR fixes**
Inability to set port range flag
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Update all script shebangs to use /usr/bin/env interpreter instead of /bin/interpreter
This is required to support systems where bash doesn't reside in /bin (such as NixOS, or the *BSD family) and allow users to specify a different interpreter version through $PATH manipulation.
https://www.cyberciti.biz/tips/finding-bash-perl-python-portably-using-env.html
```release-note
Use /usr/bin/env in all script shebangs to increase portability.
```
* Adjust hyperkube Makefile to allow HYPERKUBE_BIN to be
specified from outside
* new DOCKER_ROOT variable to specify --root-dir and associated
volume for kubelet to know where docker directory lives
* Save the docker logs when we kill the kubelet container
* Allow customized image for dockerized kubelet and build a Default
image if one is not specified
We should allow a custom build using script we have:
`VERSION="latest" REGISTRY="k8s.gcr.io" hack/dev-push-hyperkube.sh`
to be used quickly using DOCKERIZE_KUBELET. To do this we remove the
hard coded `k8s.gcr.io/kubelet` image and introduce a new environment
variable `KUBELET_IMAGE`.
Note that we are switching to hyperkube from kubelet as we have a quick
and easy script to build hyperkube image and load it into local docker
daemon. This reduces bad hacks like `docker tag
k8s.gcr.io/hyperkube-amd64:<tag> k8s.gcr.io/kubelet:latest` being used
by folks today (see 62057)
* Better cope with PID 0 when running "docker inspect" to look for
the process id for kubelet container
Automatic merge from submit-queue (batch tested with PRs 62726, 60085, 62583). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Remove podpreset in local up cluster
This was enabled in 6a7aca3129, but
shouldn't have been since the podpreset implementation is in the process
of being moved into service-catalog.
Release note:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 62445, 62768, 60633). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Local up cluster enhancements
**What this PR does / why we need it**:
Networking in a cluster brought up by hack/local-up-cluster.sh was broken because kube-proxy fails to start: connectivity to the Internet is okay, but anything involving communication among services (for example, starting cluster DNS) fails.
Debugging this took a while because there was no indication in the output of hack/local-up-cluster.sh that something had gone wrong. There were also some harmless "no such process" errors during cleanup.
The kube-proxy startup issue was fixed in parallel by PR #60431 but that fix turned out to be incomplete, so this PR uses a different approach, suppresses the harmless error messages (caused by a race) and also makes it more obvious when processes die by checking them regularly.
**Special notes for your reviewer**:
I used
KUBE_ENABLE_CLUSTER_DNS=true NET_PLUGIN=kubenet hack/local-up-cluster.sh -O
to start the cluster. Check the kube-dns pod and its logs to see how kube-dns fails to communicate with the API server.
I've left out health checking of kubelet when run under Docker because that isn't something that I have used myself yet and I wanted to send the enhancements without further delays. I can try to add that later once you agree that such a health check is useful.
**Release note**:
```release-note
fix network setup in hack/local-up-cluster.sh (https://github.com/kubernetes/kubernetes/pull/60431)
better error diagnostics in hack/local-up-cluster.sh output
```
Automatic merge from submit-queue (batch tested with PRs 58784, 62057, 62621, 62652, 62656). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Update docker run flags for containized kubelet in hack/local-up-cluster.sh
**What this PR does / why we need it**:
I'm working on https://github.com/kubernetes/kubernetes/issues/61741, and found `docker run` flags in `hack/local-up-cluster.sh` for kubelet container is too old.
- /var/lib/docker should be mount read-write, because kubelet needs to
update /etc/resolv.conf file in container, see #29378
- /var/lib/kubelet should be mount shared, because mountpoints under
this directory must be propagated to other containers
- `--pid=host` is needed to run `nsenter --mount=/rootfs/proc/1/ns/mnt -- /usr/bin/systemd-run --scope` correctly (don't know exact reason, but with `--pid=host`, it works)
- configure cluster dns for kubelet running in a container
- should propagate sub-mounts of / to kubelet for local volume
With this fix, we can start a local cluster with containerized kubelet.
```
# pull or build a hyperkube image first
docker tag k8s.gcr.io/hyperkube-amd64:<tag> k8s.gcr.io/kubelet:latest
export DOCKERIZE_KUBELET=y
./hack/local-up-cluster.sh
```
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
refer to: https://github.com/kubernetes/kubernetes-anywhere/blob/master/phase2/ignition/vanilla/kubelet.service.
**Release note**:
```release-note
NONE
```
- /var/lib/docker should be mount read-write, because kubelet needs to
update /etc/resolv.conf file in container, see #29378
- /var/lib/kubelet should be mount rslave, because mountpoints under
this directory must be propagated to other containers
- `--pid=host` is needed to run `nsenter --mount=/rootfs/proc/1/ns/mnt -- /usr/bin/systemd-run --scope` correctly
- configure cluster dns for kubelet running in a container
- should propagate sub-mounts of / to kubelet for local volume
- Use rslave to mount /var/lib/docker/ so we can see what the host
mounts under this path.
- Keep containerized kubelet flags same as non-containerized and check
kubelet is running or not.
- Use `all_kubelet_flags` to share kubelet flags.
Commit 43cb024402 replaced command line parameters with a .yaml
configuration file. But feature gates must be configured with a map in
.yaml, not with a comma-separated string as in the command line
parameters. As a result, kube-proxy failed to start and networking was
broken in the cluster.
Commit c339fc0c4f tried to fix that by moving feature gates back to
the command line, but later it was found out that the command line
parameter gets ignored when also specifying a config.
Therefore now the feature gates variable gets converted into a proper
map in the config.
This helps developers detect when something went wrong and then also
points them in the right direction by showing where the corresponding
log file is, if there is one.
For example, since commit 43cb024402 in April 2017 kube-proxy failed
to start because featureGates was not set correctly in the .yaml file,
but no-one seems to have noticed this.
Killing processes in cleanup is racing with those processes
terminating by themselves when aborting local-up-cluster.sh with
CTRL-C. That leads to unnecessary error messages:
^CCleaning up...
kill: (21592): No such process
kill: (21586): No such process
Automatic merge from submit-queue (batch tested with PRs 59027, 62333, 57661, 62086, 61584). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
local-up-cluster.sh: support preserving etcd optionally
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
**What this PR does / why we need it**:
Allow `hack/local-up-cluster.sh` to support preserve `ETCD_DIR` when `PRESERVE` is set to non-false value.
usage:
```
$ PRESERVE=true ETCD_DIR=/var/lib/etcd-kube-HEAD .... \
./hack/local-up-cluster.sh
```
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
(None)
**Special notes for your reviewer**:
(None)
**Release note**:
```release-note
None
```
Automatic merge from submit-queue (batch tested with PRs 61904, 61565, 61401, 61432, 61772). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Remove rktnetes code
**What this PR does / why we need it**:
rktnetes is scheduled to be deprecated in 1.10 (#53601). According to the deprecation policy for beta CLI and flags, we can remove the feature in 1.11.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#58721
**Special notes for your reviewer**:
**Release note**:
```release-note
Removed rknetes code, which was deprecated in 1.10.
```
/assign @yujuhong
/hold
Hold until the end of the freeze.
In nested virutalized environments, the 1 second max-time is too
low. Just bumping up WAIT_FOR_URL_API_SERVER does not work unless
we bump up the max-time too. Let us just make it configurable like
so folks can customize to their environment.
Automatic merge from submit-queue (batch tested with PRs 60980, 61273, 60811, 61021, 61367). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
prevent conformance test failure in DIND scenario
**What this PR does / why we need it**:
in the "ci-kubernetes-local-e2e" job, we are running local-up-cluster.sh
using DIND. We see failure for "HostPath should give a volume the
correct mode [Conformance]" test. The failure looks as follows:
linux mounts: path /tmp is mounted on / but it is not a shared or slave
mount
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
https://github.com/kubernetes/kubernetes/issues/61058#issuecomment-372764783
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 61124, 59537, 61235, 61258, 61114). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
remove outdated waiting apiserver comments in local-up-cluster.sh
**What this PR does / why we need it**:
Now secure port is used, remove outdated waiting for apiserver healthy comments in local-up-cluster.sh.
**Release note**:
```release-note
NONE
```
in the "ci-kubernetes-local-e2e" job, we are running local-up-cluster.sh
using DIND. We see failure for "HostPath should give a volume the
correct mode [Conformance]" test. The failure looks as follows:
linux mounts: path /tmp is mounted on / but it is not a shared or slave
mount
Local cluster is leaking memory due to mutation detector being enabled.
In addition there is no warning in the logs that this could be the
issue.
Added a log warning when this feature is enabled to make debugging this
issue easier for other cases of this.
Fixed import ordering as per @liggitt
Automatic merge from submit-queue (batch tested with PRs 59365, 60446, 60448, 55019, 60431). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Use feature-gates command line for kube-proxy
**What this PR does / why we need it**:
Avoid throwing `featureGates:` into the kube-proxy yaml, since it's
not in the right format. It should be
```
featureGates:
AllAlpha: true
```
Since FEATURES_GATES is set to `AllAlpha=true`, we can't just throw that
into the yaml file. Just pass along FEATURES_GATES in the command line
instead just like we do for all other services
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#60426
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Avoid throwing `featureGates:` into the kube-proxy yaml, since it's
not in the right format. It should be
```
featureGates:
AllAlpha: true
```
Since FEATURES_GATES is set to `AllAlpha=true`, we can't just throw that
into the yaml file. Just pass along FEATURES_GATES in the command line
instead just like we do for all other services
