Automatic merge from submit-queue
update strategic patch test for merge list of maps
Refer #26418 for more details. @janetkuo the test case is added, ptal.
Automatic merge from submit-queue
Add volume reconstruct/cleanup logic in kubelet volume manager
Currently kubelet volume management works on the concept of desired
and actual world of states. The volume manager periodically compares the
two worlds and perform volume mount/unmount and/or attach/detach
operations. When kubelet restarts, the cache of those two worlds are
gone. Although desired world can be recovered through apiserver, actual
world can not be recovered which may cause some volumes cannot be cleaned
up if their information is deleted by apiserver. This change adds the
reconstruction of the actual world by reading the pod directories from
disk. The reconstructed volume information is added to both desired
world and actual world if it cannot be found in either world. The rest
logic would be as same as before, desired world populator may clean up
the volume entry if it is no longer in apiserver, and then volume
manager should invoke unmount to clean it up.
Fixes https://github.com/kubernetes/kubernetes/issues/27653
Currently kubelet volume management works on the concept of desired
and actual world of states. The volume manager periodically compares the
two worlds and perform volume mount/unmount and/or attach/detach
operations. When kubelet restarts, the cache of those two worlds are
gone. Although desired world can be recovered through apiserver, actual
world can not be recovered which may cause some volumes cannot be cleaned
up if their information is deleted by apiserver. This change adds the
reconstruction of the actual world by reading the pod directories from
disk. The reconstructed volume information is added to both desired
world and actual world if it cannot be found in either world. The rest
logic would be as same as before, desired world populator may clean up
the volume entry if it is no longer in apiserver, and then volume
manager should invoke unmount to clean it up.
Automatic merge from submit-queue
Remove kubelet pkill dependency
Issue #26093 identified pkill as one of the dependencies of kublet
which could be worked around. Build on the code introduced for pidof
and regexp for the process(es) we need to send a signal to.
Related to #26093
We should not bailout when we get an error. We should continue
processing other files/directories. We were returning the
err passed in which was causing the processing to stop.
Fixes#30377
Automatic merge from submit-queue
[GarbageCollector] measure latency
First commit is #27600.
In e2e tests, I measure the average time an item spend in the eventQueue(~1.5 ms), dirtyQueue(~13ms), and orphanQueue(~37ms). There is no stress test in e2e yet, so the number may not be useful.
<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.kubernetes.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.kubernetes.io/reviews/kubernetes/kubernetes/28387)
<!-- Reviewable:end -->
Issue #26093 identified pkill as one of the dependencies of kublet
which could be worked around. Build on the code introduced for pidof
and regexp for the process(es) we need to send a signal to.
Related to #26093
Automatic merge from submit-queue
add metrics for workqueues
Adds prometheus metrics to work queues and enables them for the resourcequota controller. It would be easy to add this to all other workqueue based controllers and gather basic responsiveness metrics.
@kubernetes/rh-cluster-infra helps debug quota controller responsiveness problems.
<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.kubernetes.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.kubernetes.io/reviews/kubernetes/kubernetes/30296)
<!-- Reviewable:end -->
Automatic merge from submit-queue
Remove kubelet dependency on pidof
Issue #26093 identified pidof as one of the dependencies of kublet
which could be worked around. In this PR, we just look at /proc
to construct the list of pids we need for a specified process
instead of running "pidof" executable
Related to #26093
<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.kubernetes.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.kubernetes.io/reviews/kubernetes/kubernetes/30002)
<!-- Reviewable:end -->
Automatic merge from submit-queue
Cut the client repo, staging it in the main repo
Tracking issue: #28559
ref: https://github.com/kubernetes/kubernetes/pull/25978#issuecomment-232710174
This PR implements the plan a few of us came up with last week for cutting client into its own repo:
1. creating "_staging" (name is tentative) directory in the main repo, using a script to copy the client and its dependencies to this directory
2. periodically publishing the contents of this staging client to k8s.io/client-go repo
3. converting k8s components in the main repo to use the staged client. They should import the staged client as if the client were vendored. (i.e., the import line should be `import "k8s.io/client-go/<pacakge name>`). This requirement is to ease step 4.
4. In the future, removing the staging area, and vendoring the real client-go repo.
The advantage of having the staging area is that we can continuously run integration/e2e tests with the latest client repo and the latest main repo, without waiting for the client repo to be vendored back into the main repo. This staging area will exist until our test matrix is vendoring both the client and the server.
In the above plan, the tricky part is step 3. This PR achieves it by creating a symlink under ./vendor, pointing to the staging area, so packages in the main repo can refer to the client repo as if it's vendored. To prevent the godep tool from messing up the staging area, we export the staged client to GOPATH in hack/godep-save.sh so godep will think the client packages are local and won't attempt to manage ./vendor/k8s.io/client-go.
This is a POC. We'll rearrange the directory layout of the client before merge.
@thockin @lavalamp @bgrant0607 @kubernetes/sig-api-machinery
<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.kubernetes.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.kubernetes.io/reviews/kubernetes/kubernetes/29147)
<!-- Reviewable:end -->
Issue #26093 identified pidof as one of the dependencies of kublet
which could be worked around. In this PR, we just look at /proc
to construct the list of pids we need for a specified process
instead of running "pidof" executable
Related to #26093
`ex.Command()` already searches the binary in PATH, no need to manually
specify it. `pkg/util/exec` tests fail in non-conventional environments
due to this (e.g. NixOS).
Automatic merge from submit-queue
Run goimport for the whole repo
While removing GOMAXPROC and running goimports, I noticed quite a lot of other files also needed a goimport format. Didn't commit `*.generated.go`, `*.deepcopy.go` or files in `vendor`
This is more for testing if it builds.
The only strange thing here is the gopkg.in/gcfg.v1 => github.com/scalingdata/gcfg replace.
cc @jfrazelle @thockin
Automatic merge from submit-queue
Bump Libcontainer to latest head
@Random-Liu or @yujuhong Can any one of you please do a quick review.
I updated libcontainer in a previous PR but #29492 reverted those changes. This is needed for #27204.
Signed-off-by: Buddha Prakash <buddhap@google.com>
Automatic merge from submit-queue
Refactoring runner resource container linedelimiter to it's own pkg
Continuing my work ref #15634
Anyone is ok to review this fix.
Automatic merge from submit-queue
pkg/various: plug leaky time.New{Timer,Ticker}s
According to the documentation for Go package time, `time.Ticker` and
`time.Timer` are uncollectable by garbage collector finalizers. They
leak until otherwise stopped. This commit ensures that all remaining
instances are stopped upon departure from their relative scopes.
Similar efforts were incrementally done in #29439 and #29114.
```release-note
* pkg/various: plugged various time.Ticker and time.Timer leaks.
```
Automatic merge from submit-queue
pkg/util/goroutinemap: apply idiomatic Go cleanups
Package goroutinemap can be structurally simplified to be more
idiomatic, concise, and free of error potential. No structural changes
are made.
It is unconventional declare `sync.Mutex` directly as a pointerized
field in a parent structure. The `sync.Mutex` operates on pointer
receivers of itself; and by relying on that, the types that contain
those fields can be safely constructed using
https://golang.org/ref/spec#The_zero_value semantic.
The duration constants are already of type `time.Duration`, so
re-declaring that is redundant.
/CC: @saad-ali
Package goroutinemap can be structurally simplified to be more
idiomatic, concise, and free of error potential. No structural changes
are made.
It is unconventional declare `sync.Mutex` directly as a pointerized
field in a parent structure. The `sync.Mutex` operates on pointer
receivers of itself; and by relying on that, the types that contain
those fields can be safely constructed using
https://golang.org/ref/spec#The_zero_value.
The duration constants are already of type `time.Duration`, so
re-declaring that is redundant.
According to the documentation for Go package time, `time.Ticker` and
`time.Timer` are uncollectable by garbage collector finalizers. They
leak until otherwise stopped. This commit ensures that all remaining
instances are stopped upon departure from their relative scopes.
Automatic merge from submit-queue
add enhanced volume and mount logging for block devices
Fixes#24568
Adding better logging and debugging for block device volumes and the shared SafeFormatAndMount (aws, gce, flex, rbd, cinder, etc...)
Automatic merge from submit-queue
Fix Windows terminal handling
Fix some issues with Windows terminal handling with respect to TTYs that came up as part of the
code that adds support for terminal resizing.
cc @smarterclayton @sttts @csrwng
Automatic merge from submit-queue
Allow mounts to run in parallel for non-attachable volumes
This PR:
* Fixes https://github.com/kubernetes/kubernetes/issues/28616
* Enables mount volume operations to run in parallel for non-attachable volume plugins.
* Enables unmount volume operations to run in parallel for all volume plugins.
* Renames `GoRoutineMap` to `GoroutineMap`, resolving a long outstanding request from @thockin: `"Goroutine" is a noun`
Allow mount volume operations to run in parallel for non-attachable
volume plugins.
Allow unmount volume operations to run in parallel for all volume
plugins.
Automatic merge from submit-queue
Fix port range checking, port should not be greater than 65535.
When passing flag `--proxy-port-range` to kube-proxy with an invalid range which is greater than 65535, the proxy doesn't exit. That's not what we want.
Should we fix this in v1.3?
/cc @thockin @mikedanese @resouer
Before fixing:
```
root@vm:/home/paas/zxp# kube-proxy --master=172.16.1.11:8080 --logtostderr=false --log-dir=/home/user/log/kube --proxy-port-range=65536-65599 &
[6] 6671
root@vm:/home/paas/zxp# ps -ef | grep kube-proxy
root 6671 13507 0 03:48 pts/1 00:00:00 kube-proxy --master=172.16.1.11:8080 --logtostderr=false --log-dir=/home/user/log/kube --proxy-port-range=65536-65599
```
After:
```
root@vm:/home/paas/zxp# kube-proxy --master=172.16.1.11:8080 --logtostderr=false --log-dir=/home/user/log/kube --proxy-port-range=65536-65599 &
[6] 6725
root@vm:/home/paas/zxp# invalid argument "65536-65599" for --proxy-port-range=65536-65599: "65536-65599" is not a valid port range: the port range cannot be greater than 65535: 65536-65599
..............
[6]+ Exit 2 kube-proxy --master=172.16.1.11:8080 --logtostderr=false --log-dir=/home/user/log/kube --proxy-port-range=65536-65599
```
```
root@vm:/home/paas/zxp# kube-proxy --master=172.16.1.11:8080 --logtostderr=false --log-dir=/home/user/log/kube --proxy-port-range=6000-65599 &
[6] 6732
root@vm:/home/paas/zxp# invalid argument "6000-65599" for --proxy-port-range=6000-65599: "6000-65599" is not a valid port range: the port range cannot be greater than 65535: 6000-65599
..............
[6]+ Exit 2 kube-proxy --master=172.16.1.11:8080 --logtostderr=false --log-dir=/home/user/log/kube --proxy-port-range=6000-65599
```
Automatic merge from submit-queue
Reexport term.IsTerminal
Reexport term.IsTerminal so downstream consumers (e.g. OpenShift) can use it.
@smarterclayton @sttts
Automatic merge from submit-queue
Support terminal resizing for exec/attach/run
```release-note
Add support for terminal resizing for exec, attach, and run. Note that for Docker, exec sessions
inherit the environment from the primary process, so if the container was created with tty=false,
that means the exec session's TERM variable will default to "dumb". Users can override this by
setting TERM=xterm (or whatever is appropriate) to get the correct "smart" terminal behavior.
```
Fixes#13585
Add support for terminal resizing for exec, attach, and run. Note that for Docker, exec sessions
inherit the environment from the primary process, so if the container was created with tty=false,
that means the exec session's TERM variable will default to "dumb". Users can override this by
setting TERM=xterm (or whatever is appropriate) to get the correct "smart" terminal behavior.
Automatic merge from submit-queue
controller: wait for synced old replica sets on Recreate
Partially fixes https://github.com/kubernetes/kubernetes/issues/27362
Any other work on it should be handled in the replica set level (and/or kubelet if it's required)
@kubernetes/deployment PTAL
Automatic merge from submit-queue
Add meta field to predicate signature to avoid computing the same things multiple times
This PR only uses it to avoid computing QOS of a pod for every node from scratch.
Ref #28590
Automatic merge from submit-queue
Fixes bad heuristic when calling "tc show" to check interface
`tc` sometimes returns stuff that has more than 12 words in its response. The heuristic is bad, but this at least fixes the case when `tc` is returning too much.
Fixes#28571.
This drives most of the logic of deep-copy generation from tags like:
// +deepcopy-gen=package
..rather than hardcoded lists of packages. This will make it possible to
subsequently generate code ONLY for packages that need it *right now*, rather
than all of them always.
Also remove pkgs that really do not need deep-copies (no symbols used
anywhere).
His PR cam during the middle of this development cycle, and it was easier to
burn it down and recreate it than try to patch it into an existing series and
re-test every assumption. This behavior will be re-introduced in subsequent
commits.
Automatic merge from submit-queue
Validation: Make validation func return error strings
Part of an ongoing series of validation cleanups.
This centralizes the error strings next to the code that checks the error conditions. Future commits will refine the messages further and provide more utility validators.
I'm OK if this doesn't go into 1.2, but I am tired of rebasing :) I suggest commit-by-commit review, which should go pretty quickly. This was largely mechanical.
<!-- Reviewable:start -->
---
This change is [<img src="http://reviewable.k8s.io/review_button.svg" height="35" align="absmiddle" alt="Reviewable"/>](http://reviewable.k8s.io/reviews/kubernetes/kubernetes/21240)
<!-- Reviewable:end -->
Automatic merge from submit-queue
Cleanup third party (pt 2)
Move forked-and-hacked golang code to the forked/ directory. Remove ast/build/parse code that is now in stdlib. Remove unused shell2junit
Automatic merge from submit-queue
dedup workqueue requeuing
Updates `workqueue.AddAfter` to only perform the add for the earliest requested requeue operation. An earlier time inserts in the earlier slot and removes the old one. A later time is ignored.
When using this conjunction with an `AddRateLimited` method, you get charged for the additional retry even though you're only queue once.
This keeps requeues from multiplying for every add.
@liggitt
Automatic merge from submit-queue
Reorganize volume controllers and manager
* Move both PV and attach/detach volume controllers to `controllers/volume` (closes#26222)
* Rename `kubelet/volume` to `kubelet/volumemanager`
* Add/update OWNER files
Automatic merge from submit-queue
Add MinReadySeconds to rolling updater
Add MinReadySeconds support to RollingUpdater that allows to specify the number of seconds to wait on top of the pod is "ready" because its readiness probe passed.
Automatic merge from submit-queue
Fixed goroutinemap race on Wait()
sync.WaitGroup produces data races when a GoroutineMap is empty and Wait() and Run() are called at the same time.
From sync.WaitGroup:
> Note that calls with a positive delta that occur when the counter is zero must happen before a Wait.
Fixes#28128
Note that this issue affects only PersistentVolume unit tests.
@saad-ali, PTAL
Automatic merge from submit-queue
Enable HTTP2 by default
Update to enable HTTP2 by default, with the option to disable.
This is a continuation of #25280 for the 1.4 release. This should provide ample time for vetting.
/cc @krousey
sync.WaitGroup produces data races when a GoroutineMap is empty and Wait() and
Run() are called at the same time.
From sync.WaitGroup:
Note that calls with a positive delta that occur when the counter is zero
must happen before a Wait.
Fixes#28128
Automatic merge from submit-queue
Proportionally scale paused and rolling deployments
Enable paused and rolling deployments to be proportionally scaled.
Also have cleanup policy work for paused deployments.
Fixes#20853Fixes#20966Fixes#20754
@bgrant0607 @janetkuo @ironcladlou @nikhiljindal
<!-- Reviewable:start -->
---
This change is [<img src="http://reviewable.k8s.io/review_button.svg" height="35" align="absmiddle" alt="Reviewable"/>](http://reviewable.k8s.io/reviews/kubernetes/kubernetes/20273)
<!-- Reviewable:end -->
Automatic merge from submit-queue
GCE provider: Create TargetPool with 200 instances, then update with rest
GCE provider: Create TargetPool with 200 instances, then update with rest
Tested with 2000 nodes, this actually meets the GCE API specifications (which is nutty). Previous PR (#25178) was based on a mistaken understanding of a poorly documented set of limitations, and even poorer testing, for which I am embarassed.
Also includes the revert of #25178 (review commits separately).
[]()
Modify attach/detach controller to keep track of volumes to report
attached in Node VolumeToAttach status.
Modify kubelet volume manager to wait for volume to show up in Node
VolumeToAttach status.
Implement exponential backoff for errors in volume manager and attach
detach controller
Automatic merge from submit-queue
Fix bug in isLikelyNotMountPoint function
In nsenter_mount.go/isLikelyNotMountPoint function, the returned output
from findmnt command misses the last letter. Modify the code to use
String.contains instead of string matching. fixes#26421fixes#25056fixes#22911
- improve restoreInternal implementation in iptables
- add SetStdin and SetStdout functions to Cmd interface
- modify kubelet/prober and some tests in order to work with Cmd interface
In nsenter_mount.go/isLikelyNotMountPoint function, the returned output
from findmnt command misses the last letter. Modify the code to make sure
that output has the full target path. fix#26421#25056#22911
Automatic merge from submit-queue
Listing pods only once when getting pods for RS in deployment
Fixes#26834
1. Avoid ranging over RSes and then `List` pods of each RS. Instead, `List` pods of the deployment once, and then filter pods of each RS.
2. Avoid using clientset to `List` pods in deployment controller. Use podStore instead. (TODO in some functions because the unit tests don't have podStore.)
@kubernetes/deployment
[]()
Automatic merge from submit-queue
rkt: Fix incomplete selinux context string when the option is partial.
Fix "EmptyDir" e2e tests failures caused by #https://github.com/kubernetes/kubernetes/pull/24901
As mentioned in https://github.com/kubernetes/kubernetes/pull/24901#discussion_r61372312
We should apply the selinux context of the rkt data directory (/var/lib/rkt) when users do not specify all the selinux options.
Due to my fault, the change was missed during rebase, thus caused the regression.
After applying this PR, the e2e tests passed.
```
$ go run hack/e2e.go -v -test --test_args="--ginkgo.dryRun=false --ginkgo.focus=EmptyDir"
...
Ran 19 of 313 Specs in 199.319 seconds
SUCCESS! -- 19 Passed | 0 Failed | 0 Pending | 294 Skipped PASS
```
BTW, the test is removed because the `--no-overlay=true` flag will only be there on non-coreos distro.
cc @euank @kubernetes/sig-node
Automatic merge from submit-queue
Add specific error type for "operation already exists" error.
PersistentVolume controller needs to know why scheduling a new operation has failed - if the operation was already running or some other error happened.
Automatic merge from submit-queue
Sort revisions in rollout history as integers
Previously keys were sorted as strings, thus it was possible to see such order as 1, 10, 2, 3, 4, 5.
fixes: #25788
Automatic merge from submit-queue
Make UnsafeConversion fast by inlining copies
Not ready yet (need to add a copy to "safe" conversion and add mutation tests to roundtrip api/serialization_test).
Cuts another 10% off decode and encode.
Automatic merge from submit-queue
Increase goroutinemap unit test timeouts.
50ms is flaky in Jenkins. This makes the test to take at least 0.5s to check that things that should block and wait for something really do block and wait (was 100ms before).
Fixes#25825
Previously keys were sorted as strings, thus it was possible
to see such order as 1, 10, 2, 3, 4, 5.
Ints64 helper implemented in util/slice module to sort []int64
Automatic merge from submit-queue
Cache Webhook Authentication responses
Add a simple LRU cache w/ 2 minute TTL to the webhook authenticator.
Kubectl is a little spammy, w/ >= 4 API requests per command. This also prevents a single unauthenticated user from being able to DOS the remote authenticator.
Automatic merge from submit-queue
Only expose top N images in `NodeStatus`
Fix#25209
Sorted the image and only pick set top 50 sized images in node status.
cc @vishh
Automatic merge from submit-queue
Fixes panic on round tripper when TLS under a proxy
When under a proxy with a valid cert from a trusted authority, the `SpdyRoundTripper` will likely not have a `*tls.Config` (no cert verification nor `InsecureSkipVerify` happened), which will result in a panic. So we have to create a new `*tls.Config` to be able to create a TLS client right after. If `RootCAs` in that new config is nil, the system pool will be used.
@ncdc PTAL
[]()
PersistentVolume controller needs to know why scheduling a new operation has
failed - if the operation was already running or some other error happened.
Automatic merge from submit-queue
Make IsQualifiedName return error strings
Part of the larger validation PR, broken out for easier review and merge.
@lavalamp FYI, but I know you're swamped, too.
Automatic merge from submit-queue
Rate limitting requeue
I think this will do what I want in almost every case. I'm going to try it out for a single writer quota evaluation.
`NewRateLimitingQueue(NewControllerRateLimiter(qps, burst))` gives an `AddRateLimited(item)` that backs off based on the max of a bucket and exponential backoff per item.
@liggitt Want to see if the interface works for you too?
<!-- Reviewable:start -->
---
This change is [<img src="http://reviewable.k8s.io/review_button.svg" height="35" align="absmiddle" alt="Reviewable"/>](http://reviewable.k8s.io/reviews/kubernetes/kubernetes/24052)
<!-- Reviewable:end -->
Automatic merge from submit-queue
Upgrades tests use chaosmonkey package and ServiceTestJig
Introduce the `chaosmonkey` e2e package for doing disruptive testing (e.g. upgrade testing) more easily.
- [x] `chaosmonkey` package
- [x] migrate upgrade tests to `chaosmonkey` (using WIP `serviceJig`)
- [x] migrate upgrade tests to use `ServiceTestJig` and `chaosmonkey`
Deferred:
- [ ] make `ServiceTestJig` implement `chaosmonkey.Interface`
- [ ] migrate disruptive services tests to use `ServiceTestJig` and `chaosmonkey`
This provides the extensible framework for #15131. We should now easily be able to add tests (e.g. #6084, #23189).
This is a rewrite of #22446.
cc @mikedanese @quinton-hoole @roberthbailey
Assigning to @thockin, who wrote `ServiceTestJig`.
<!-- Reviewable:start -->
---
This change is [<img src="http://reviewable.k8s.io/review_button.svg" height="35" align="absmiddle" alt="Reviewable"/>](http://reviewable.k8s.io/reviews/kubernetes/kubernetes/24014)
<!-- Reviewable:end -->
Automatic merge from submit-queue
Display line number on JSON errors
Related issue: https://github.com/kubernetes/kubernetes/issues/12231
This PR will introduce line numbers for all JSON errors in the CLI:
(this is existing error reporting for YAML)
```console
$ kubectl create -f broken.yaml
yaml: line 8: mapping values are not allowed in this context
```
(this is error reporting proposed in this PR for JSON)
```console
$ kubectl create -f broken.json
json: line 35: invalid character '{' after object key
```
(and this is the current reporting:)
```console
$ kubectl create -f broken.json
invalid character '{' after object key
```
[]()
Automatic merge from submit-queue
Add data structure for managing go routines by name
This PR introduces a data structure for managing go routines by name. It prevents the creation of new go routines if an existing go routine with the same name exists. This will enable parallelization of the designs in https://github.com/kubernetes/kubernetes/issues/20262 and https://github.com/kubernetes/kubernetes/issues/21931 with sufficient protection to prevent starting multiple operations on the same volume.
Automatic merge from submit-queue
Kubelet: Cleanup with new engine api
Finish step 2 of #23563
This PR:
1) Cleanup go-dockerclient reference in the code.
2) Bump up the engine-api version.
3) Cleanup the code with new engine-api.
Fixes#24076.
Fixes#23809.
/cc @yujuhong
Automatic merge from submit-queue
Frame decoder was checking cap(), not len()
Resulted in bytes being missing from the streaming decoder. Update both
parts.
Required for #24789
@wojtek-t or @lavalamp while he's OOO
Automatic merge from submit-queue
Unify termination signal handling across platforms
The signals are used to ensure proper execution of cleaning up actions to restore a terminal in:
- [pkg/kubectl/cmd/attach.go#L163-L226](b8d000853e/pkg/kubectl/cmd/attach.go (L163-L226))
- [pkg/kubectl/cmd/util/editor/editor.go#L112-L137](b8d000853e/pkg/kubectl/cmd/util/editor/editor.go (L112-L137))
All supported platforms can handle the same set of signals we're interested in, thus we don't need build contraints to use a set of signals on Linux, while restricting ourselves to only SIGINT on Darwin and Windows.
According to the documentation of os/signal, similar to SIGINT and SIGTERM, SIGHUP causes the program to exit, therefore add it to the list of handled signals.
The fist commit is part of #23643.
fix https://github.com/kubernetes/kubernetes/issues/24717
If kubelet root-dir is a symlink, the pod's cinder volume dir can't be
umounted even after pod is deleted.
This patch reads target path of symlink before comparing with entries in
proc mounts.
- All supported platforms can handle the same set of signals we're
interested in, thus we don't need build contraints to use a set of
signals on Linux, while restricting ourselves to only SIGINT on Darwin
and Windows.
- According to the documentation of os/signal, similar to SIGINT and
SIGTERM, SIGHUP causes the program to exit, therefore add it to the
list of handled signals.
Signed-off-by: Rodolfo Carvalho <rcarvalh@redhat.com>
Automatic merge from submit-queue
Generated clients can return their RESTClients, RESTClient can return its RateLimiter
cc @lavalamp @krousey @wojtek-t @smarterclayton @timothysc
Ref. #22421
Automatic merge from submit-queue
Fix use of docker removed ParseRepositoryTag() function
Docker has removed the ParseRepositoryTag() function in
leading to failures using the kubernetes Go client API.
Failure:
```
../k8s.io/kubernetes/pkg/util/parsers/parsers.go:30: undefined: parsers.ParseRepositoryTag
```
Docker has removed the ParseRepositoryTag() function in
leading to failures using the kubernetes Go client API.
Lets use github.com/docker/distribution reference.ParseNamed()
instead.
Failure:
../k8s.io/kubernetes/pkg/util/parsers/parsers.go:30: undefined: parsers.ParseRepositoryTag
Add tests to watch behavior in both protocols (http and websocket)
against all 3 media types. Adopt the
`application/vnd.kubernetes.protobuf;stream=watch` media type for the
content that comes back from a watch call so that it can be
distinguished from a Status result.
Automatic merge from submit-queue
Remove requirement that Endpoints IPs be IPv4
Signed-off-by: André Martins <aanm90@gmail.com>
Release Note: The `Endpoints` API object now allows IPv6 addresses to be stored. Other components of the system are not ready for IPv6 yet, and many cloud providers are not IPv6 compatible, but installations that use their own controller logic can now store v6 endpoints.
Automatic merge from submit-queue
Do not throw creation errors for containers that fail immediately after being started
Fixes (hopefully) #23607
cc @dchen1107
Automatic merge from submit-queue
add a delayed queueing option to the workqueue
Adds delayed requeuing to the workqueue so that I requeue an item to be retried at some later time in my controller with a series of backoff rules. It lets me have the best of the retryManager and the work queue de-duping. Tracking failures and backoffs is on the caller
@smarterclayton @pweil- this would help us move to using the informer everywhere and de-duping at that level.
Automatic merge from submit-queue
Fixed mounting with containerized kubelet
`NsenterMounter.IsLikelyNotMountPoint()` should return `ErrNotExist` when the
checked directory does not exists - the regular mounted does this and
some volume plugins depend on this behavior.
See for example: https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/aws_ebs/aws_util.go#L72
Automatic merge from submit-queue
Implement a streaming serializer for watch
Changeover watch to use streaming serialization. Properly version the
watch objects. Implement simple framing for JSON and Protobuf (but not
YAML).
@wojtek-t @lavalamp
Automatic merge from submit-queue
Preserve int data when unmarshaling
There are several places we use `json.Unmarshal` into an unstructured map (StrategicMergePatch, UnstructuredJSONScheme, many others).
In this scenario, the json package converts all numbers to float64. This exposes many of the int64 fields in our API types to corruption when the unstructured map is marshalled back to json.
A simple example is a pod with an `"activeDeadlineSeconds": 1000000`. Trying to use `kubectl label`, `annotate`, `patch`, etc results in that int64 being converted to a float64, submitted to the server, and the server rejecting it with an error about "cannot unmarshal number 1e+6 into Go value of type int64"
The json package provides a way to defer conversion of numbers (`json.Decoder#UseNumber`), but does not actually do conversions to int or float. This PR makes use of that feature, and post-processes the unmarshalled map to convert json.Number objects to either int64 or float64 values
Automatic merge from submit-queue
Additional go vet fixes
Mostly:
- pass lock by value
- bad syntax for struct tag value
- example functions not formatted properly
Automatic merge from submit-queue
Move typed clients into clientset folder
Move typed clients from `pkg/client/typed/` to `pkg/client/clientset_generated/${clientset_name}/typed`.
The first commit changes the client-gen, the last commit updates the doc, other commits are just moving things around.
@lavalamp @krousey
Automatic merge from submit-queue
Use correct home directory on Windows
As of now the `kubectl` config mechanism that saves/loads `~/.kube/config` is broken on Windows, saving the config file in the *current* directory instead of the user's *home* dir. This happens because most Windows don't respect the `HOME` environment variable.
This PR changes the config file loading mechanism to use the recommended way to detect the user home on Windows (`HOMEDRIVE`+`HOMEPATH` or `USERPROFILE`), and adds a migration for users that might be currently relying on existing config files.
Automatic merge from submit-queue
Refactor streaming code to support interop testing
Refactor exec/attach/port forward client and server code to better
support interop testing of different client and server subprotocol
versions.
Fixes#16119
os/signal.Stop() does not actually close the channel, which means
the channel is blocked. Not serious because we don't use interrupt
protection on the server, just the CLI (which can afford 1 leaked
goroutine).
Kubelet was not able to mount volumes when running inside a container and
using nsenter mounter,
NsenterMounter.IsLikelyNotMountPoint() should return ErrNotExist when the
checked directory does not exists as the regular mounted does this and
some volume plugins depend on this behavior.
Currently attach and the editor do not share the same logic for saving
and restoring the terminal, and are not suitable for nesting (when the
caller wants to create something, attach, and then delete something when
the attach is over). This commit moves the interrupt protection logic
to a util package and supports nesting interrupt handlers.
This ensures that the call to close the underlying streaming connection will execute immediately,
instead of waiting for all streams to gracefully shut down.
Due to rounding down for maxUnavailable, we may end up with rolling updates
that have zero surge and unavailable pods something that 1) is not allowed
as per validation, 2) blocks updates. If we end up in such a situation
set maxUnavailable to 1 on the theory that surge might not work due to
quota.
Had to move other things around too to avoid a weird api ->
cloudprovider dependency.
Also adding fixes per code reviews.
(This is a squash of the previously approved commits)
The message as it is framed right now does not make any sense for the
end users of our system. It might even lead to confusion. So this is
attempt to make the error message less confusing.
Update the Deployments' API types, defaulting code, conversions, helpers
and validation to use ReplicaSets instead of ReplicationControllers and
LabelSelector instead of map[string]string for selectors.
Also update the Deployment controller, registry, kubectl subcommands,
client listers package and e2e tests to use ReplicaSets and
LabelSelector for Deployments.
In order to synchronize the current state of Kubernetes's objects (e.g. pods, containers, etc.),
periodic synch loops are run. When there is a lot of objects to synchronize with,
loops increase communication traffic. At some point when all the traffic interfere cpu usage curve
hits the roof causing 100% cpu utilization.
To distribute the traffic in time, some sync loops can jitter their period in each loop
and help to flatten the curve.
This commit adds JitterUntil function with jitterFactor parameter.
JitterUntil generalizes Until which is a special case for jitterFactor being zero.
Add a mutex to guard SetUpAt() and TearDownAt() calls - they should not
run in parallel. There is a race in these calls when there are two pods
using the same volume, one of them is dying and the other one starting.
TearDownAt() checks that a volume is not needed by any pods and detaches the
volume. It does so by counting how many times is the volume mounted
(GetMountRefs() call below).
When SetUpAt() of the starting pod already attached the volume and did not mount
it yet, TearDownAt() of the dying pod will detach it - GetMountRefs() does not
count with this volume.
These two threads run in parallel:
dying pod.TearDownAt("myVolume") starting pod.SetUpAt("myVolume")
| |
| AttachDisk("myVolume")
refs, err := mount.GetMountRefs() |
Unmount("myDir") |
if refs == 1 { |
| | Mount("myVolume", "myDir")
| | |
| DetachDisk("myVolume") |
| start containers - OOPS! The volume is detached!
|
finish the pod cleanup
Also, add some logs to cinder plugin for easier debugging in the future, add
a test and update the fake mounter to know about bind mounts.
Add a recognizer that is capable of sniffing content type from data by
asking each serializer to try to decode - this is for a "universal
decoder/deserializer" which can be used by client logic.
Add codec factory, which provides the core primitives for content type
negotiation. Codec factories depend only on schemes, serializers, and
groupversion pairs.
Used like:
var pod *api.Pod
err := client.RetryOnConflict(client.DefaultBackoff, func() (err error) {
pod, err = c.Pods("mynamespace").UpdateStatus(update)
return
})
// err may be conflict
We can either fix it here or at every callsite. Every callsite is
currently using this method incorrectly.
Signed-off-by: Mike Danese <mikedanese@google.com>
I took a hard look at error output and played until I was happier. This now
prints JSON for structs in the error, rather than go's format.
Also made the error message easier to read.
Fixed tests.
Before this change we have a mish-mash of ways to pass field names around for
error generation. Sometimes string fieldnames, sometimes .Prefix(), sometimes
neither, often wrong names or not indexed when it should be.
Instead of that mess, this is part one of a couple of commits that will make it
more strongly typed and hopefully encourage correct behavior. At least you
will have to think about field names, which is better than nothing.
It turned out to be really hard to do this incrementally.
All external types that are not int64 are now marked as int32,
including
IntOrString. Prober is now int32 (43 years should be enough of an initial
probe time for anyone).
Did not change the metadata fields for now.
It's hard to manage connection buffers in a deterministic test, ensure
that the error is always consistent and the output is always a subset
of the input
Contains the following fixes for Windows users of kubectl edit:
* Defaults to notepad as the default Windows editor
* Uses CRLF line endings
* Ensures a file lock is freed
Add streaming subprotocol negotiation for exec, attach, and port
forwarding. Restore previous (buggy) exec functionality as an
unspecified/unversioned subprotocol so newer kubectl clients can work
against 1.0.x kubelets.
The current executor structure is too dependent on client.Request
and client.Config. In order to do an attach from the server, it needs
to be possible to create an Executor from crypto/tls#TLSConfig and to
bypassing having a client.Request.
Changes:
* remotecommand.spdyExecutor - handles upgrading a request to SPDY and getting a connection
* remotecommand.NewAttach / New - moved to exec / portforward / attach since they handle requests
* Remove request.Upgrade() - it's too coupled to SPDY, and can live with the spdyExecutor
* Add request.VersionedParams(runtime.Object, runtime.ObjectConvertor) to handle object -> query transform
Not all clients and systems can support SPDY protocols. This commit adds
support for two new websocket protocols, one to handle streaming of pod
logs from a pod, and the other to allow exec to be tunneled over
websocket.
Browser support for chunked encoding is still poor, and web consoles
that wish to show pod logs may need to make compromises to display the
output. The /pods/<name>/log endpoint now supports websocket upgrade to
the 'binary.k8s.io' subprotocol, which sends chunks of logs as binary to
the client. Messages are written as logs are streamed from the container
daemon, so flushing should be unaffected.
Browser support for raw communication over SDPY is not possible, and
some languages lack libraries for it and HTTP/2. The Kubelet supports
upgrade to WebSocket instead of SPDY, and will multiplex STDOUT/IN/ERR
over websockets by prepending each binary message with a single byte
representing the channel (0 for IN, 1 for OUT, and 2 for ERR). Because
framing on WebSockets suffers from head-of-line blocking, clients and
other server code should ensure that no particular stream blocks. An
alternative subprotocol 'base64.channel.k8s.io' base64 encodes the body
and uses '0'-'9' to represent the channel for ease of use in browsers.
Correct port-forward data copying logic so that the server closes its
half of the data stream when socat exits, and the client closes its half
of the data stream when it finishes writing.
Modify the client to wait for both copies (client->server,
server->client) to finish before it unblocks.
Fix race condition in the Kubelet's handling of incoming port forward
streams. Have the client generate a connectionID header to be used to
associate the error and data streams for a single connection, instead of
assuming that streams n and n+1 go together. Attempt to generate a
pseudo connectionID in the server in the event the connectionID header
isn't present (older clients); this is a best-effort approach that only
really works with 1 connection at a time, whereas multiple concurrent
connections will only work reliably with a newer client that is
generating connectionID.
Have poller() send to the channel once, immediately, before the ticker
starts. This way, Poll, PollInfinite, and WaitFor will check the
condition immediately, instead of waiting for the poller's interval to
elapse once before doing the initial condition check.
Increase the supported controls on pod logging. Add validaiton to pod
log options. Ensure the Kubelet is using a consistent, structured way to
process pod log arguments.
Add ?sinceSeconds=<durationInSeconds>, &sinceTime=<RFC3339>, ?timestamps=<bool>,
?tailLines=<number>, and ?limitBytes=<number>
The `file` command used here to check whether a device is formatted is not
available for CoreOS. The effect is that the mounter tries to mount an
unformatted volume which fails. This makes it quite tedious to use persistent
volumes in CoreOS.
This patch replaces the `file` command with `lsblk` which is available in
CoreOS. I checked that it's also available on RHEL, Debian, Ubuntu and SLES.
A lot of packages use StringSet, but they don't use anything else from
the util package. Moving StringSet into another package will shrink
their dependency trees significantly.
If stdin is noninteractive, the io.Copy from stdin to remoteStdin will
unblock when it finishes reading from stdin. In this case, make sure to
close remoteStdin so the server knows the client won't be sending any
more data. This ensures that the remote process terminates. For example:
echo foo | kubectl exec -i <pod> -- cat
Without this change, the `cat` process never terminates and `kubectl
exec` hangs.
Fix interactive exec sessions hanging after you type 'exit'.
Add e2e test to cover noninteractive stdin: `echo a | kubectl exec -i <pod>
cat`
Add e2e test to cover psuedo-interactive stdin: `kubectl exec -i <pod> bash`
Prep for sending multiple data frames over multiple streams in remote command
test, which is more likely to find flakes (requires bump of spdystream
once an issue with the frame worker queues not being fully drained when
a goaway frame is received).
Use iptables --wait (if available) to avoid race conditions with
util.iptables failing if it tries to modify the tables at the same
time as another process.
Also, reorganize the code a bit in preparation for checking for
another flag as well. And, if semver.NewVersion() returns an error, it
means there's a bug in the code somewhere (we should only ever be
passing it valid version strings), so just log that error rather than
returning it to the caller.
The GCE PD plugin uses safe_format_and_mount found on standard GCE images:
https://github.com/GoogleCloudPlatform/compute-image-packages/blob/master/google-startup-scripts/usr/share/google/safe_format_and_mount
On custom images where this is not available pods fail to format and
mount GCE PDs. This patch uses linux utilities in a similar way to the
safe_format_and_mount script to format and mount the GCE PD and AWS EBC
devices. That is first attempt a mount. If mount fails try to use file to
investigate the device. If 'file' fails to get any information about
the device and simply returns "data" then assume the device is not
formatted and format it and attempt to mount it again.
Signed-off-by: Sami Wagiaalla <swagiaal@redhat.com>
The API server will refuse to start if a link-local IP address is selected, however, ChooseNetworkInterface can still select link-local network routes. This causes a startup failure on platforms like DigitalOcean that use a link-local route to fetch metadata.
Signed-off-by: Christian Stewart <christian@paral.in>
With older iptables binary, kube-proxy generates duplicate
iptables rules in NAT table every few seconds.
This fixes the problem by properly unquoting && parsing
older iptables-save output.
This turned out to be a little convoluted, but is needed because deleting an ELB on AWS
is a painful UX - it won't have the same endpoint when it is recreated.
Also started splitting the provider into files, but only for new functions (so far!)
IsLikelyNotMountPoint determines if a directory is not a mountpoint.
It is fast but not necessarily ALWAYS correct. If the path is in fact
a bind mount from one part of a mount to another it will not be detected.
mkdir /tmp/a /tmp/b; mount --bin /tmp/a /tmp/b; IsLikelyNotMountPoint("/tmp/b")
will return true. When in fact /tmp/b is a mount point. So this patch
renames the function and switches it from a positive to a negative (I
could think of a good positive name). This should make future users of
this function aware that it isn't quite perfect, but probably good
enough.
Since pflag can handle net.IPNet arguements use that code. This means
that our code no longer has casts back and forth and just natively uses
net.IPNet.
pflag can handle IP addresses so use the pflag code instead of doing it
ourselves. This means our code just uses net.IP and we don't have all of
the useless casting back and forth!
Fixes#10842
All issues for types that use "extra lines" for extended information. Two issues fixed: (1) When namespaces are listed an extra column isn't inserted for extra lines and (2) trailing tabs aren't inserted when label columns are specified.
This code should probably move to a more explicit model of putting data into "cells".
The test for this hits https://github.com/golang/go/issues/6416 and so I introduced a "LineDelimiter" writer filter to make white space more visible.
PR #10643 Started adding the dns names for the kubernetes master to self
sign certs which were created. The kubelet uses this same code, and thus
the kubelet cert started saying it was valid for these name as well.
While hardless, the kubelet cert shouldn't claim to be these things. So
make the caller explicitly list both their ip and dns subject alt names.
A cert from GCE shows:
- IP Address:23.236.49.122
- IP Address:10.0.0.1
- DNS:kubernetes,
- DNS:kubernetes.default
- DNS:kubernetes.default.svc
- DNS:kubernetes.default.svc.cluster.local
- DNS:e2e-test-zml-master
A similarly configured self signed cert shows:
- IP Address:23.236.49.122
- IP Address:10.0.0.1
- DNS:kubernetes
- DNS:kubernetes.default
- DNS:kubernetes.default.svc
So we are missing the fqdn kubernetes.default.svc.cluster.local. The
apiserver does not even know the fqdn! it's defined entirely by the
kubelet! We also do not have the cluster name certificate. This may be
--cluster-name= argument to the apiserver but will take a bit more
research.
Some downstream components want to use a StringSet in their internal
objects, but leaving util.empty private means that generated conversions
cannot create constructors generically for those objects. This makes
Empty public until such a time as the generator supports private types.
This works on gcloud (where the user is dynamically created by the tool),
but doesn't hold on other clouds (e.g. AWS).
The function in pkg/util now takes a user arg, and it is called only
from the e2e tests, which now check for env-var KUBE_SSH_USER, and then
fall back to the existing behaviour of env-var USER.
I am using this from Jenkins by directly setting the env-var:
export KUBE_SSH_USER=jenkins
...
hack/jenkins/e2e.sh
The new approach avoids building a complete parse tree, instead using the lower level token stream. Doing so removes the need for creating "missing" HTML elements, reducing the collateral changes to the rewritten HTML.
Resolves: #9766
Refactor GetNodeHostIP into pkg/util/node (instead of pkg/util to break import cycle).
Include internalIP in gce NodeAddresses. Remove NodeLegacyHostIP
A service with a NodePort set will listen on that port, on every node.
This is both handy for some load balancers (AWS ELB) and for people
that want to expose a service without using a load balancer.
This change appends the full hostname to the mirror pod name (instead of taking
the first token) so that if the hostname is overriden, we'd not be creating
unncessary name conflicts. An example would be that a user overrides the
hostnames to be "127.0.0.1" and "127.0.0.2", and both of them were resolved to
"127" for the mirror pod name suffix.
Also, because `uname -n` could return a FQDN or not, this change takes only
the first token of it as the hostname for consistency.
Remove creation of stream from TestServeExecInContainerIdleTimeout as
it's not necessary to very idle timeout.
Increase stream creation and ack timeouts to 30 seconds.
Fixes#5628
This commit adds support to core resources to enable deferred deletion
of resources. Clients may optionally specify a time period after which
resources must be deleted via an object sent with their DELETE. That
object may define an optional grace period in seconds, or allow the
default "preferred" value for a resource to be used. Once the object
is marked as pending deletion, the deletionTimestamp field will be set
and an etcd TTL will be in place.
Clients should assume resources that have deletionTimestamp set will
be deleted at some point in the future. Other changes will come later
to enable graceful deletion on a per resource basis.
# *** ERROR: *** docs are out of sync between cli and markdown
# run hack/run-gendocs.sh > docs/kubectl.md to regenerate
#
# Your commit will be aborted unless you regenerate docs.
COMMIT_BLOCKED_ON_GENDOCS
Hostname behavior across operating systems is inconsistent (Macs can
have uppercase host names, so can some other systems). In general,
always strings.ToLower(os.Hostname()).
Currently, kubelet silently ignores pods that caused host port conflict. This
commit surfaces the error by recording an event.
It also makes sure that kubelet iterates through the pods in the order of the
creation timestamp, which ensures that pods created later are ignored on
conflict.
This is the start of an uber-binary that can morph into any server. Eventually we'll want this to be able to launch multiple servers from a single command line.
It would be better if Kubernetes could be instrumented to report panics
and handled but unreturned errors in controller loops to a remote
service. This commit introduces settable handlers for HandleCrash and a new
HandleError that can be overriden to report errors to a remote service
for analysis. HandleError() in particular is for control loops that do
not return an error, and so there is no ability to report those errors
other than in log files.
ReplicationController is provided as an example.
This makes glog flags such as --logtostderr and --alsologtostderr keep
working without requiring an explicit =true.
Tested by building kubelet and invoking it with --logtostderr. Also
tested a few non-boolean flags to ensure they weren't affected.
Break up the monolithic volumes code in kubelet into very small individual
modules with a well-defined interface. Move them all into their own packages
and beef up testing along the way.
This exposes the proper v1beta3 API endpoint when the user specifies
the --runtime_config=api/v1beta3 argument to the apiserver. v1beta3
is still considered experimental and subject to change.
--runtime_config is a map of string keys and values, that can be
specified by providing
--runtime_config=a=b,b=c,d,e
Only the key must be specified, the value can be omitted.
Enables v1beta3 in hack/local-up-cluster.sh and hack/test-cmd.sh
There are quite a few 'composite literal uses unkeyed fields' errors that I have kept out of this patch.
And there's a couple where vet just seems confused. These are the easiest ones.
After this DNS is resolvable from the host, if the DNS server is targetted
explicitly. This does NOT add the cluster DNS to the host's resolv.conf. That
is a larger problem, with distro-specific tie-ins and circular deps.
The iptables args list needs to include all fields as they are eventually spit
out by iptables-save. This is because some systems do not support the
'iptables -C' arg, and so fall back on parsing iptables-save output. If this
does not match, it will not pass the check. For example: adding the /32 on
the destination IP arg is not strictly required, but causes this list to not
match the final iptables-save output. This is fragile and I hope one day we
can stop supporting such old iptables versions.
People were misusing EncodeJSON in tests when they should be using
runtime.EncodeOrDie(testapi.Codec(), obj). Removing the potential
for cutting self on sharp objects.
This makes iptables an injectable dependency. The test for this started off
simple, but got a bit out of hand. It's starting to become a mocking
framework.
* Allows consumers to provide their own transports for common cases.
* Supports KUBE_API_VERSION on test cases for controlling which
api version they test against
* Provides a common flag registration method for CLIs that need
to connect to an API server (to avoid duplicating flags)
* Ensures errors are properly returned by the server
* Add a Context field to client.Config
Also rename some to other names that make better reading. There are still a
bunch of "make" functions but they do things like assemble a string from parts
or build an array of things. It seemed that "make" there seemed fine. "New"
is for "constructors".
Because time.Time doesn't work correctly with our YAML package, it is necessary
to introduce a type, util.Time, which serializes correctly to JSON and YAML.
Eventually we would like timestamping to cut across storage implementations;
for now, we set it in each storage.
Specifying an API type as IntOrString will allow JSON and YAML to accept
either ints or strings with the same name. For example, port names or
numbers.
1) imported glog to third_party (previous commit)
2) add support for third_party/update.sh to update just one pkg
3) search-and-replace:
s/log.Printf/glog.Infof/
s/log.Print/glog.Info/
s/log.Fatalf/glog.Fatalf/
s/log.Fatal/glog.Fatal/
4) convert glog.Info.*, err into glog.Error*
Adds some util interfaces to logging and calls them from each cmd, which
will set the default log output to write to glog. Pass glog-wrapped
Loggers to etcd for logging.
Log files will go to /tmp - we should probably follow this up with a
default log dir for each cmd.
The glog lib is sort of weak in that it only flushes every 30 seconds, so
we spin up our own flushing goroutine.
