Automatic merge from submit-queue
Don't return an error if a file doesn't exist for IsPathDevice(...)
Fixes https://github.com/kubernetes/kubernetes/issues/30455
@saad-ali @thockin fyi, since linux devices and storage.
It is common in constrained circumstances to prefer an empty string
result from JSONPath templates for missing keys over an error. Several
other implementations provide this (the canonical JS and PHP, as well as
the Java implementation). This also mirrors gotemplate, which allows
Options("missingkey=zero").
Added simple check and simple test case.
Automatic merge from submit-queue
Add AppArmor feature gate
Add option to disable AppArmor via a feature gate. This PR treats AppArmor as Beta, and thus depends on https://github.com/kubernetes/kubernetes/pull/31471 (I will remove `do-not-merge` once that merges).
Note that disabling AppArmor means that pods with AppArmor annotations will be rejected in validation. It does not mean that the components act as though AppArmor was never implemented. This is by design, because we want to make it difficult to accidentally run a Pod with an AppArmor annotation without AppArmor protection.
/cc @dchen1107
Automatic merge from submit-queue
Fix hang/websocket timeout when streaming container log with no content
When streaming and following a container log, no response headers are sent from the kubelet `containerLogs` endpoint until the first byte of content is written to the log. This propagates back to the API server, which also will not send response headers until it gets response headers from the kubelet. That includes upgrade headers, which means a websocket connection upgrade is not performed and can time out.
To recreate, create a busybox pod that runs `/bin/sh -c 'sleep 30 && echo foo && sleep 10'`
As soon as the pod starts, query the kubelet API:
```
curl -N -k -v 'https://<node>:10250/containerLogs/<ns>/<pod>/<container>?follow=true&limitBytes=100'
```
or the master API:
```
curl -N -k -v 'http://<master>:8080/api/v1/<ns>/pods/<pod>/log?follow=true&limitBytes=100'
```
In both cases, notice that the response headers are not sent until the first byte of log content is available.
This PR:
* does a 0-byte write prior to handing off to the container runtime stream copy. That commits the response header, even if the subsequent copy blocks waiting for the first byte of content from the log.
* fixes a bug with the "ping" frame sent to websocket streams, which was not respecting the requested protocol (it was sending a binary frame to a websocket that requested a base64 text protocol)
* fixes a bug in the limitwriter, which was not propagating 0-length writes, even before the writer's limit was reached
Automatic merge from submit-queue
Improve godoc for goroutinemap
Improves the godoc of goroutinemap; found while preparing to use this type in another PR.
@saad-ali
Automatic merge from submit-queue
Fixed integer overflow bug in rate limiter.
```release-note
Fix overflow issue in controller-manager rate limiter
```
This PR fixes a bug in the delayed work-queue used by some controllers.
The integer overflow bug would previously cause hotlooping behavior after a few failures
as `time.Duration(..)` on values larger than MaxInt64 behaves unpredictably, and
after a certain value returns 0 always.
cc @bprashanth @pwittrock
Automatic merge from submit-queue
Revert revert 30090 with fix
This reverts #31297 (which originally reverted #30090) and applies a fix to stop the fd leak that was exposed by #30090.
Automatic merge from submit-queue
Avoid sorting lists when unnecessary
I've seen ThreadSafeMap::List consuming ~30% of whole CPU usage, spending the whole time in sorting (while it is in fact completely unneded).
Automatic merge from submit-queue
Add kubelet --network-plugin-mtu flag for MTU selection
* Add network-plugin-mtu option which lets us pass down a MTU to a network provider (currently processed by kubenet)
* Add a test, and thus make sysctl testable
Automatic merge from submit-queue
[Kubelet] Optionally consume configuration from <node-name> named config maps
This extends the Kubelet to check the API server for new node-specific config, and exit when it finds said new config.
/cc @kubernetes/sig-node @mikedanese @timstclair @vishh
**Release note**:
```
Extends Kubelet with Alpha Dynamic Kubelet Configuration. Please note that this alpha feature does not currently work with cloud provider auto-detection.
```
Automatic merge from submit-queue
Unset https_proxy before roundtripper_test
When running `hack/test-go.sh`, if the testing env is behind a https proxy, roundtripper_test will fail randomly.
After `unset https_proxy`, the testing works well. So, add a comment to be a troubleshooting tip.
Fail info:
```
--- FAIL: TestRoundTripAndNewConnection (0.12s)
roundtripper_test.go:319: proxied http->http: shouldError=false, got true: Get http://127.0.0.1:46711: unexpected EOF
FAIL
FAIL k8s.io/kubernetes/pkg/util/httpstream/spdy 0.148s
```
```
--- FAIL: TestRoundTripAndNewConnection (0.12s)
roundtripper_test.go:319: proxied https with auth (valid hostname + RootCAs) -> http: shouldError=false, got true: Get http://127.0.0.1:41028: unexpected EOF
FAIL
FAIL k8s.io/kubernetes/pkg/util/httpstream/spdy 0.146s
```
Automatic merge from submit-queue
Make 'allAlpha' a special feature gate
Rather than making all caller check both allAlpha and their own flag, make `allAlpha` set all of the alpha gates explicitly.
This is hard to test because of the globalness. I will follow this commit with a new one to add some way to test, but I wanted to float this design
Automatic merge from submit-queue
Implement TLS bootstrap for kubelet using `--experimental-bootstrap-kubeconfig` (2nd take)
Ref kubernetes/features#43 (comment)
cc @gtank @philips @mikedanese @aaronlevy @liggitt @deads2k @errordeveloper @justinsb
Continue on the older PR https://github.com/kubernetes/kubernetes/pull/30094 as there are too many comments on that one and it's not loadable now.
Automatic merge from submit-queue
Fixes#30886
This PR fixes https://github.com/kubernetes/kubernetes/issues/30886
```
make WHAT=pkg/kubelet
+++ [0818 17:03:21] Generating bindata:
/Users/jscheuermann/inovex/workspace/kubernetes-clone/test/e2e/framework/gobindata_util.go
+++ [0818 17:03:22] Building the toolchain targets:
k8s.io/kubernetes/hack/cmd/teststale
+++ [0818 17:03:22] Building go targets for darwin/amd64:
pkg/kubelet
```
Add --bootstrap-kubeconfig flag to kubelet. If the flag is non-empty
and --kubeconfig doesn't exist, then the kubelet will use the bootstrap
kubeconfig to create rest client and generate certificate signing request
to request a client cert from API server.
Once succeeds, the result cert will be written down to
--cert-dir/kubelet-client.crt, and the kubeconfig will be populated with
certfile, keyfile path pointing to the result certificate file, key file.
(The key file is generated before creating the CSR).
Automatic merge from submit-queue
Add annotations to the PodSecurityPolicy Provider interface
@pweil- is this what you were thinking in terms of API changes? I really like to avoid functions with more than 2 return values, but couldn't think of a cleaner approach in this case.
Automatic merge from submit-queue
use Reader.ReadLine instead of bufio.Scanner to support bigger yaml
@smarterclayton ptal. Also refer #19603#23125 for more details.
Automatic merge from submit-queue
update strategic patch test for merge list of maps
Refer #26418 for more details. @janetkuo the test case is added, ptal.
Automatic merge from submit-queue
Add volume reconstruct/cleanup logic in kubelet volume manager
Currently kubelet volume management works on the concept of desired
and actual world of states. The volume manager periodically compares the
two worlds and perform volume mount/unmount and/or attach/detach
operations. When kubelet restarts, the cache of those two worlds are
gone. Although desired world can be recovered through apiserver, actual
world can not be recovered which may cause some volumes cannot be cleaned
up if their information is deleted by apiserver. This change adds the
reconstruction of the actual world by reading the pod directories from
disk. The reconstructed volume information is added to both desired
world and actual world if it cannot be found in either world. The rest
logic would be as same as before, desired world populator may clean up
the volume entry if it is no longer in apiserver, and then volume
manager should invoke unmount to clean it up.
Fixes https://github.com/kubernetes/kubernetes/issues/27653
Currently kubelet volume management works on the concept of desired
and actual world of states. The volume manager periodically compares the
two worlds and perform volume mount/unmount and/or attach/detach
operations. When kubelet restarts, the cache of those two worlds are
gone. Although desired world can be recovered through apiserver, actual
world can not be recovered which may cause some volumes cannot be cleaned
up if their information is deleted by apiserver. This change adds the
reconstruction of the actual world by reading the pod directories from
disk. The reconstructed volume information is added to both desired
world and actual world if it cannot be found in either world. The rest
logic would be as same as before, desired world populator may clean up
the volume entry if it is no longer in apiserver, and then volume
manager should invoke unmount to clean it up.
Automatic merge from submit-queue
Remove kubelet pkill dependency
Issue #26093 identified pkill as one of the dependencies of kublet
which could be worked around. Build on the code introduced for pidof
and regexp for the process(es) we need to send a signal to.
Related to #26093
We should not bailout when we get an error. We should continue
processing other files/directories. We were returning the
err passed in which was causing the processing to stop.
Fixes#30377
Automatic merge from submit-queue
[GarbageCollector] measure latency
First commit is #27600.
In e2e tests, I measure the average time an item spend in the eventQueue(~1.5 ms), dirtyQueue(~13ms), and orphanQueue(~37ms). There is no stress test in e2e yet, so the number may not be useful.
<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.kubernetes.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.kubernetes.io/reviews/kubernetes/kubernetes/28387)
<!-- Reviewable:end -->
Issue #26093 identified pkill as one of the dependencies of kublet
which could be worked around. Build on the code introduced for pidof
and regexp for the process(es) we need to send a signal to.
Related to #26093
Automatic merge from submit-queue
add metrics for workqueues
Adds prometheus metrics to work queues and enables them for the resourcequota controller. It would be easy to add this to all other workqueue based controllers and gather basic responsiveness metrics.
@kubernetes/rh-cluster-infra helps debug quota controller responsiveness problems.
<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.kubernetes.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.kubernetes.io/reviews/kubernetes/kubernetes/30296)
<!-- Reviewable:end -->
Automatic merge from submit-queue
Remove kubelet dependency on pidof
Issue #26093 identified pidof as one of the dependencies of kublet
which could be worked around. In this PR, we just look at /proc
to construct the list of pids we need for a specified process
instead of running "pidof" executable
Related to #26093
<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.kubernetes.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.kubernetes.io/reviews/kubernetes/kubernetes/30002)
<!-- Reviewable:end -->
Automatic merge from submit-queue
Cut the client repo, staging it in the main repo
Tracking issue: #28559
ref: https://github.com/kubernetes/kubernetes/pull/25978#issuecomment-232710174
This PR implements the plan a few of us came up with last week for cutting client into its own repo:
1. creating "_staging" (name is tentative) directory in the main repo, using a script to copy the client and its dependencies to this directory
2. periodically publishing the contents of this staging client to k8s.io/client-go repo
3. converting k8s components in the main repo to use the staged client. They should import the staged client as if the client were vendored. (i.e., the import line should be `import "k8s.io/client-go/<pacakge name>`). This requirement is to ease step 4.
4. In the future, removing the staging area, and vendoring the real client-go repo.
The advantage of having the staging area is that we can continuously run integration/e2e tests with the latest client repo and the latest main repo, without waiting for the client repo to be vendored back into the main repo. This staging area will exist until our test matrix is vendoring both the client and the server.
In the above plan, the tricky part is step 3. This PR achieves it by creating a symlink under ./vendor, pointing to the staging area, so packages in the main repo can refer to the client repo as if it's vendored. To prevent the godep tool from messing up the staging area, we export the staged client to GOPATH in hack/godep-save.sh so godep will think the client packages are local and won't attempt to manage ./vendor/k8s.io/client-go.
This is a POC. We'll rearrange the directory layout of the client before merge.
@thockin @lavalamp @bgrant0607 @kubernetes/sig-api-machinery
<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.kubernetes.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.kubernetes.io/reviews/kubernetes/kubernetes/29147)
<!-- Reviewable:end -->
Issue #26093 identified pidof as one of the dependencies of kublet
which could be worked around. In this PR, we just look at /proc
to construct the list of pids we need for a specified process
instead of running "pidof" executable
Related to #26093
`ex.Command()` already searches the binary in PATH, no need to manually
specify it. `pkg/util/exec` tests fail in non-conventional environments
due to this (e.g. NixOS).
Automatic merge from submit-queue
Run goimport for the whole repo
While removing GOMAXPROC and running goimports, I noticed quite a lot of other files also needed a goimport format. Didn't commit `*.generated.go`, `*.deepcopy.go` or files in `vendor`
This is more for testing if it builds.
The only strange thing here is the gopkg.in/gcfg.v1 => github.com/scalingdata/gcfg replace.
cc @jfrazelle @thockin
Automatic merge from submit-queue
Bump Libcontainer to latest head
@Random-Liu or @yujuhong Can any one of you please do a quick review.
I updated libcontainer in a previous PR but #29492 reverted those changes. This is needed for #27204.
Signed-off-by: Buddha Prakash <buddhap@google.com>
Automatic merge from submit-queue
Refactoring runner resource container linedelimiter to it's own pkg
Continuing my work ref #15634
Anyone is ok to review this fix.
Automatic merge from submit-queue
pkg/various: plug leaky time.New{Timer,Ticker}s
According to the documentation for Go package time, `time.Ticker` and
`time.Timer` are uncollectable by garbage collector finalizers. They
leak until otherwise stopped. This commit ensures that all remaining
instances are stopped upon departure from their relative scopes.
Similar efforts were incrementally done in #29439 and #29114.
```release-note
* pkg/various: plugged various time.Ticker and time.Timer leaks.
```
Automatic merge from submit-queue
pkg/util/goroutinemap: apply idiomatic Go cleanups
Package goroutinemap can be structurally simplified to be more
idiomatic, concise, and free of error potential. No structural changes
are made.
It is unconventional declare `sync.Mutex` directly as a pointerized
field in a parent structure. The `sync.Mutex` operates on pointer
receivers of itself; and by relying on that, the types that contain
those fields can be safely constructed using
https://golang.org/ref/spec#The_zero_value semantic.
The duration constants are already of type `time.Duration`, so
re-declaring that is redundant.
/CC: @saad-ali
Package goroutinemap can be structurally simplified to be more
idiomatic, concise, and free of error potential. No structural changes
are made.
It is unconventional declare `sync.Mutex` directly as a pointerized
field in a parent structure. The `sync.Mutex` operates on pointer
receivers of itself; and by relying on that, the types that contain
those fields can be safely constructed using
https://golang.org/ref/spec#The_zero_value.
The duration constants are already of type `time.Duration`, so
re-declaring that is redundant.
According to the documentation for Go package time, `time.Ticker` and
`time.Timer` are uncollectable by garbage collector finalizers. They
leak until otherwise stopped. This commit ensures that all remaining
instances are stopped upon departure from their relative scopes.
Automatic merge from submit-queue
add enhanced volume and mount logging for block devices
Fixes#24568
Adding better logging and debugging for block device volumes and the shared SafeFormatAndMount (aws, gce, flex, rbd, cinder, etc...)
Automatic merge from submit-queue
Fix Windows terminal handling
Fix some issues with Windows terminal handling with respect to TTYs that came up as part of the
code that adds support for terminal resizing.
cc @smarterclayton @sttts @csrwng
Automatic merge from submit-queue
Allow mounts to run in parallel for non-attachable volumes
This PR:
* Fixes https://github.com/kubernetes/kubernetes/issues/28616
* Enables mount volume operations to run in parallel for non-attachable volume plugins.
* Enables unmount volume operations to run in parallel for all volume plugins.
* Renames `GoRoutineMap` to `GoroutineMap`, resolving a long outstanding request from @thockin: `"Goroutine" is a noun`
Allow mount volume operations to run in parallel for non-attachable
volume plugins.
Allow unmount volume operations to run in parallel for all volume
plugins.
Automatic merge from submit-queue
Fix port range checking, port should not be greater than 65535.
When passing flag `--proxy-port-range` to kube-proxy with an invalid range which is greater than 65535, the proxy doesn't exit. That's not what we want.
Should we fix this in v1.3?
/cc @thockin @mikedanese @resouer
Before fixing:
```
root@vm:/home/paas/zxp# kube-proxy --master=172.16.1.11:8080 --logtostderr=false --log-dir=/home/user/log/kube --proxy-port-range=65536-65599 &
[6] 6671
root@vm:/home/paas/zxp# ps -ef | grep kube-proxy
root 6671 13507 0 03:48 pts/1 00:00:00 kube-proxy --master=172.16.1.11:8080 --logtostderr=false --log-dir=/home/user/log/kube --proxy-port-range=65536-65599
```
After:
```
root@vm:/home/paas/zxp# kube-proxy --master=172.16.1.11:8080 --logtostderr=false --log-dir=/home/user/log/kube --proxy-port-range=65536-65599 &
[6] 6725
root@vm:/home/paas/zxp# invalid argument "65536-65599" for --proxy-port-range=65536-65599: "65536-65599" is not a valid port range: the port range cannot be greater than 65535: 65536-65599
..............
[6]+ Exit 2 kube-proxy --master=172.16.1.11:8080 --logtostderr=false --log-dir=/home/user/log/kube --proxy-port-range=65536-65599
```
```
root@vm:/home/paas/zxp# kube-proxy --master=172.16.1.11:8080 --logtostderr=false --log-dir=/home/user/log/kube --proxy-port-range=6000-65599 &
[6] 6732
root@vm:/home/paas/zxp# invalid argument "6000-65599" for --proxy-port-range=6000-65599: "6000-65599" is not a valid port range: the port range cannot be greater than 65535: 6000-65599
..............
[6]+ Exit 2 kube-proxy --master=172.16.1.11:8080 --logtostderr=false --log-dir=/home/user/log/kube --proxy-port-range=6000-65599
```
Automatic merge from submit-queue
Reexport term.IsTerminal
Reexport term.IsTerminal so downstream consumers (e.g. OpenShift) can use it.
@smarterclayton @sttts
Automatic merge from submit-queue
Support terminal resizing for exec/attach/run
```release-note
Add support for terminal resizing for exec, attach, and run. Note that for Docker, exec sessions
inherit the environment from the primary process, so if the container was created with tty=false,
that means the exec session's TERM variable will default to "dumb". Users can override this by
setting TERM=xterm (or whatever is appropriate) to get the correct "smart" terminal behavior.
```
Fixes#13585
Add support for terminal resizing for exec, attach, and run. Note that for Docker, exec sessions
inherit the environment from the primary process, so if the container was created with tty=false,
that means the exec session's TERM variable will default to "dumb". Users can override this by
setting TERM=xterm (or whatever is appropriate) to get the correct "smart" terminal behavior.
Automatic merge from submit-queue
controller: wait for synced old replica sets on Recreate
Partially fixes https://github.com/kubernetes/kubernetes/issues/27362
Any other work on it should be handled in the replica set level (and/or kubelet if it's required)
@kubernetes/deployment PTAL
Automatic merge from submit-queue
Add meta field to predicate signature to avoid computing the same things multiple times
This PR only uses it to avoid computing QOS of a pod for every node from scratch.
Ref #28590
Automatic merge from submit-queue
Fixes bad heuristic when calling "tc show" to check interface
`tc` sometimes returns stuff that has more than 12 words in its response. The heuristic is bad, but this at least fixes the case when `tc` is returning too much.
Fixes#28571.
This drives most of the logic of deep-copy generation from tags like:
// +deepcopy-gen=package
..rather than hardcoded lists of packages. This will make it possible to
subsequently generate code ONLY for packages that need it *right now*, rather
than all of them always.
Also remove pkgs that really do not need deep-copies (no symbols used
anywhere).
His PR cam during the middle of this development cycle, and it was easier to
burn it down and recreate it than try to patch it into an existing series and
re-test every assumption. This behavior will be re-introduced in subsequent
commits.
Automatic merge from submit-queue
Validation: Make validation func return error strings
Part of an ongoing series of validation cleanups.
This centralizes the error strings next to the code that checks the error conditions. Future commits will refine the messages further and provide more utility validators.
I'm OK if this doesn't go into 1.2, but I am tired of rebasing :) I suggest commit-by-commit review, which should go pretty quickly. This was largely mechanical.
<!-- Reviewable:start -->
---
This change is [<img src="http://reviewable.k8s.io/review_button.svg" height="35" align="absmiddle" alt="Reviewable"/>](http://reviewable.k8s.io/reviews/kubernetes/kubernetes/21240)
<!-- Reviewable:end -->
Automatic merge from submit-queue
Cleanup third party (pt 2)
Move forked-and-hacked golang code to the forked/ directory. Remove ast/build/parse code that is now in stdlib. Remove unused shell2junit
Automatic merge from submit-queue
dedup workqueue requeuing
Updates `workqueue.AddAfter` to only perform the add for the earliest requested requeue operation. An earlier time inserts in the earlier slot and removes the old one. A later time is ignored.
When using this conjunction with an `AddRateLimited` method, you get charged for the additional retry even though you're only queue once.
This keeps requeues from multiplying for every add.
@liggitt
Automatic merge from submit-queue
Reorganize volume controllers and manager
* Move both PV and attach/detach volume controllers to `controllers/volume` (closes#26222)
* Rename `kubelet/volume` to `kubelet/volumemanager`
* Add/update OWNER files
Automatic merge from submit-queue
Add MinReadySeconds to rolling updater
Add MinReadySeconds support to RollingUpdater that allows to specify the number of seconds to wait on top of the pod is "ready" because its readiness probe passed.
Automatic merge from submit-queue
Fixed goroutinemap race on Wait()
sync.WaitGroup produces data races when a GoroutineMap is empty and Wait() and Run() are called at the same time.
From sync.WaitGroup:
> Note that calls with a positive delta that occur when the counter is zero must happen before a Wait.
Fixes#28128
Note that this issue affects only PersistentVolume unit tests.
@saad-ali, PTAL
Automatic merge from submit-queue
Enable HTTP2 by default
Update to enable HTTP2 by default, with the option to disable.
This is a continuation of #25280 for the 1.4 release. This should provide ample time for vetting.
/cc @krousey
sync.WaitGroup produces data races when a GoroutineMap is empty and Wait() and
Run() are called at the same time.
From sync.WaitGroup:
Note that calls with a positive delta that occur when the counter is zero
must happen before a Wait.
Fixes#28128
Automatic merge from submit-queue
Proportionally scale paused and rolling deployments
Enable paused and rolling deployments to be proportionally scaled.
Also have cleanup policy work for paused deployments.
Fixes#20853Fixes#20966Fixes#20754
@bgrant0607 @janetkuo @ironcladlou @nikhiljindal
<!-- Reviewable:start -->
---
This change is [<img src="http://reviewable.k8s.io/review_button.svg" height="35" align="absmiddle" alt="Reviewable"/>](http://reviewable.k8s.io/reviews/kubernetes/kubernetes/20273)
<!-- Reviewable:end -->
Automatic merge from submit-queue
GCE provider: Create TargetPool with 200 instances, then update with rest
GCE provider: Create TargetPool with 200 instances, then update with rest
Tested with 2000 nodes, this actually meets the GCE API specifications (which is nutty). Previous PR (#25178) was based on a mistaken understanding of a poorly documented set of limitations, and even poorer testing, for which I am embarassed.
Also includes the revert of #25178 (review commits separately).
[]()
Modify attach/detach controller to keep track of volumes to report
attached in Node VolumeToAttach status.
Modify kubelet volume manager to wait for volume to show up in Node
VolumeToAttach status.
Implement exponential backoff for errors in volume manager and attach
detach controller
Automatic merge from submit-queue
Fix bug in isLikelyNotMountPoint function
In nsenter_mount.go/isLikelyNotMountPoint function, the returned output
from findmnt command misses the last letter. Modify the code to use
String.contains instead of string matching. fixes#26421fixes#25056fixes#22911
- improve restoreInternal implementation in iptables
- add SetStdin and SetStdout functions to Cmd interface
- modify kubelet/prober and some tests in order to work with Cmd interface
In nsenter_mount.go/isLikelyNotMountPoint function, the returned output
from findmnt command misses the last letter. Modify the code to make sure
that output has the full target path. fix#26421#25056#22911
Automatic merge from submit-queue
Listing pods only once when getting pods for RS in deployment
Fixes#26834
1. Avoid ranging over RSes and then `List` pods of each RS. Instead, `List` pods of the deployment once, and then filter pods of each RS.
2. Avoid using clientset to `List` pods in deployment controller. Use podStore instead. (TODO in some functions because the unit tests don't have podStore.)
@kubernetes/deployment
[]()
Automatic merge from submit-queue
rkt: Fix incomplete selinux context string when the option is partial.
Fix "EmptyDir" e2e tests failures caused by #https://github.com/kubernetes/kubernetes/pull/24901
As mentioned in https://github.com/kubernetes/kubernetes/pull/24901#discussion_r61372312
We should apply the selinux context of the rkt data directory (/var/lib/rkt) when users do not specify all the selinux options.
Due to my fault, the change was missed during rebase, thus caused the regression.
After applying this PR, the e2e tests passed.
```
$ go run hack/e2e.go -v -test --test_args="--ginkgo.dryRun=false --ginkgo.focus=EmptyDir"
...
Ran 19 of 313 Specs in 199.319 seconds
SUCCESS! -- 19 Passed | 0 Failed | 0 Pending | 294 Skipped PASS
```
BTW, the test is removed because the `--no-overlay=true` flag will only be there on non-coreos distro.
cc @euank @kubernetes/sig-node
Automatic merge from submit-queue
Add specific error type for "operation already exists" error.
PersistentVolume controller needs to know why scheduling a new operation has failed - if the operation was already running or some other error happened.
Automatic merge from submit-queue
Sort revisions in rollout history as integers
Previously keys were sorted as strings, thus it was possible to see such order as 1, 10, 2, 3, 4, 5.
fixes: #25788
Automatic merge from submit-queue
Make UnsafeConversion fast by inlining copies
Not ready yet (need to add a copy to "safe" conversion and add mutation tests to roundtrip api/serialization_test).
Cuts another 10% off decode and encode.
Automatic merge from submit-queue
Increase goroutinemap unit test timeouts.
50ms is flaky in Jenkins. This makes the test to take at least 0.5s to check that things that should block and wait for something really do block and wait (was 100ms before).
Fixes#25825
Previously keys were sorted as strings, thus it was possible
to see such order as 1, 10, 2, 3, 4, 5.
Ints64 helper implemented in util/slice module to sort []int64
Automatic merge from submit-queue
Cache Webhook Authentication responses
Add a simple LRU cache w/ 2 minute TTL to the webhook authenticator.
Kubectl is a little spammy, w/ >= 4 API requests per command. This also prevents a single unauthenticated user from being able to DOS the remote authenticator.
Automatic merge from submit-queue
Only expose top N images in `NodeStatus`
Fix#25209
Sorted the image and only pick set top 50 sized images in node status.
cc @vishh
Automatic merge from submit-queue
Fixes panic on round tripper when TLS under a proxy
When under a proxy with a valid cert from a trusted authority, the `SpdyRoundTripper` will likely not have a `*tls.Config` (no cert verification nor `InsecureSkipVerify` happened), which will result in a panic. So we have to create a new `*tls.Config` to be able to create a TLS client right after. If `RootCAs` in that new config is nil, the system pool will be used.
@ncdc PTAL
[]()
PersistentVolume controller needs to know why scheduling a new operation has
failed - if the operation was already running or some other error happened.
Automatic merge from submit-queue
Make IsQualifiedName return error strings
Part of the larger validation PR, broken out for easier review and merge.
@lavalamp FYI, but I know you're swamped, too.
Automatic merge from submit-queue
Rate limitting requeue
I think this will do what I want in almost every case. I'm going to try it out for a single writer quota evaluation.
`NewRateLimitingQueue(NewControllerRateLimiter(qps, burst))` gives an `AddRateLimited(item)` that backs off based on the max of a bucket and exponential backoff per item.
@liggitt Want to see if the interface works for you too?
<!-- Reviewable:start -->
---
This change is [<img src="http://reviewable.k8s.io/review_button.svg" height="35" align="absmiddle" alt="Reviewable"/>](http://reviewable.k8s.io/reviews/kubernetes/kubernetes/24052)
<!-- Reviewable:end -->
Automatic merge from submit-queue
Upgrades tests use chaosmonkey package and ServiceTestJig
Introduce the `chaosmonkey` e2e package for doing disruptive testing (e.g. upgrade testing) more easily.
- [x] `chaosmonkey` package
- [x] migrate upgrade tests to `chaosmonkey` (using WIP `serviceJig`)
- [x] migrate upgrade tests to use `ServiceTestJig` and `chaosmonkey`
Deferred:
- [ ] make `ServiceTestJig` implement `chaosmonkey.Interface`
- [ ] migrate disruptive services tests to use `ServiceTestJig` and `chaosmonkey`
This provides the extensible framework for #15131. We should now easily be able to add tests (e.g. #6084, #23189).
This is a rewrite of #22446.
cc @mikedanese @quinton-hoole @roberthbailey
Assigning to @thockin, who wrote `ServiceTestJig`.
<!-- Reviewable:start -->
---
This change is [<img src="http://reviewable.k8s.io/review_button.svg" height="35" align="absmiddle" alt="Reviewable"/>](http://reviewable.k8s.io/reviews/kubernetes/kubernetes/24014)
<!-- Reviewable:end -->
Automatic merge from submit-queue
Display line number on JSON errors
Related issue: https://github.com/kubernetes/kubernetes/issues/12231
This PR will introduce line numbers for all JSON errors in the CLI:
(this is existing error reporting for YAML)
```console
$ kubectl create -f broken.yaml
yaml: line 8: mapping values are not allowed in this context
```
(this is error reporting proposed in this PR for JSON)
```console
$ kubectl create -f broken.json
json: line 35: invalid character '{' after object key
```
(and this is the current reporting:)
```console
$ kubectl create -f broken.json
invalid character '{' after object key
```
[]()
Automatic merge from submit-queue
Add data structure for managing go routines by name
This PR introduces a data structure for managing go routines by name. It prevents the creation of new go routines if an existing go routine with the same name exists. This will enable parallelization of the designs in https://github.com/kubernetes/kubernetes/issues/20262 and https://github.com/kubernetes/kubernetes/issues/21931 with sufficient protection to prevent starting multiple operations on the same volume.
Automatic merge from submit-queue
Kubelet: Cleanup with new engine api
Finish step 2 of #23563
This PR:
1) Cleanup go-dockerclient reference in the code.
2) Bump up the engine-api version.
3) Cleanup the code with new engine-api.
Fixes#24076.
Fixes#23809.
/cc @yujuhong
Automatic merge from submit-queue
Frame decoder was checking cap(), not len()
Resulted in bytes being missing from the streaming decoder. Update both
parts.
Required for #24789
@wojtek-t or @lavalamp while he's OOO
Automatic merge from submit-queue
Unify termination signal handling across platforms
The signals are used to ensure proper execution of cleaning up actions to restore a terminal in:
- [pkg/kubectl/cmd/attach.go#L163-L226](b8d000853e/pkg/kubectl/cmd/attach.go (L163-L226))
- [pkg/kubectl/cmd/util/editor/editor.go#L112-L137](b8d000853e/pkg/kubectl/cmd/util/editor/editor.go (L112-L137))
All supported platforms can handle the same set of signals we're interested in, thus we don't need build contraints to use a set of signals on Linux, while restricting ourselves to only SIGINT on Darwin and Windows.
According to the documentation of os/signal, similar to SIGINT and SIGTERM, SIGHUP causes the program to exit, therefore add it to the list of handled signals.
The fist commit is part of #23643.
fix https://github.com/kubernetes/kubernetes/issues/24717
If kubelet root-dir is a symlink, the pod's cinder volume dir can't be
umounted even after pod is deleted.
This patch reads target path of symlink before comparing with entries in
proc mounts.
- All supported platforms can handle the same set of signals we're
interested in, thus we don't need build contraints to use a set of
signals on Linux, while restricting ourselves to only SIGINT on Darwin
and Windows.
- According to the documentation of os/signal, similar to SIGINT and
SIGTERM, SIGHUP causes the program to exit, therefore add it to the
list of handled signals.
Signed-off-by: Rodolfo Carvalho <rcarvalh@redhat.com>
Automatic merge from submit-queue
Generated clients can return their RESTClients, RESTClient can return its RateLimiter
cc @lavalamp @krousey @wojtek-t @smarterclayton @timothysc
Ref. #22421
Automatic merge from submit-queue
Fix use of docker removed ParseRepositoryTag() function
Docker has removed the ParseRepositoryTag() function in
leading to failures using the kubernetes Go client API.
Failure:
```
../k8s.io/kubernetes/pkg/util/parsers/parsers.go:30: undefined: parsers.ParseRepositoryTag
```
Docker has removed the ParseRepositoryTag() function in
leading to failures using the kubernetes Go client API.
Lets use github.com/docker/distribution reference.ParseNamed()
instead.
Failure:
../k8s.io/kubernetes/pkg/util/parsers/parsers.go:30: undefined: parsers.ParseRepositoryTag
Add tests to watch behavior in both protocols (http and websocket)
against all 3 media types. Adopt the
`application/vnd.kubernetes.protobuf;stream=watch` media type for the
content that comes back from a watch call so that it can be
distinguished from a Status result.
Automatic merge from submit-queue
Remove requirement that Endpoints IPs be IPv4
Signed-off-by: André Martins <aanm90@gmail.com>
Release Note: The `Endpoints` API object now allows IPv6 addresses to be stored. Other components of the system are not ready for IPv6 yet, and many cloud providers are not IPv6 compatible, but installations that use their own controller logic can now store v6 endpoints.
Automatic merge from submit-queue
Do not throw creation errors for containers that fail immediately after being started
Fixes (hopefully) #23607
cc @dchen1107
Automatic merge from submit-queue
add a delayed queueing option to the workqueue
Adds delayed requeuing to the workqueue so that I requeue an item to be retried at some later time in my controller with a series of backoff rules. It lets me have the best of the retryManager and the work queue de-duping. Tracking failures and backoffs is on the caller
@smarterclayton @pweil- this would help us move to using the informer everywhere and de-duping at that level.
Automatic merge from submit-queue
Fixed mounting with containerized kubelet
`NsenterMounter.IsLikelyNotMountPoint()` should return `ErrNotExist` when the
checked directory does not exists - the regular mounted does this and
some volume plugins depend on this behavior.
See for example: https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/aws_ebs/aws_util.go#L72
Automatic merge from submit-queue
Implement a streaming serializer for watch
Changeover watch to use streaming serialization. Properly version the
watch objects. Implement simple framing for JSON and Protobuf (but not
YAML).
@wojtek-t @lavalamp
Automatic merge from submit-queue
Preserve int data when unmarshaling
There are several places we use `json.Unmarshal` into an unstructured map (StrategicMergePatch, UnstructuredJSONScheme, many others).
In this scenario, the json package converts all numbers to float64. This exposes many of the int64 fields in our API types to corruption when the unstructured map is marshalled back to json.
A simple example is a pod with an `"activeDeadlineSeconds": 1000000`. Trying to use `kubectl label`, `annotate`, `patch`, etc results in that int64 being converted to a float64, submitted to the server, and the server rejecting it with an error about "cannot unmarshal number 1e+6 into Go value of type int64"
The json package provides a way to defer conversion of numbers (`json.Decoder#UseNumber`), but does not actually do conversions to int or float. This PR makes use of that feature, and post-processes the unmarshalled map to convert json.Number objects to either int64 or float64 values
Automatic merge from submit-queue
Additional go vet fixes
Mostly:
- pass lock by value
- bad syntax for struct tag value
- example functions not formatted properly
Automatic merge from submit-queue
Move typed clients into clientset folder
Move typed clients from `pkg/client/typed/` to `pkg/client/clientset_generated/${clientset_name}/typed`.
The first commit changes the client-gen, the last commit updates the doc, other commits are just moving things around.
@lavalamp @krousey
Automatic merge from submit-queue
Use correct home directory on Windows
As of now the `kubectl` config mechanism that saves/loads `~/.kube/config` is broken on Windows, saving the config file in the *current* directory instead of the user's *home* dir. This happens because most Windows don't respect the `HOME` environment variable.
This PR changes the config file loading mechanism to use the recommended way to detect the user home on Windows (`HOMEDRIVE`+`HOMEPATH` or `USERPROFILE`), and adds a migration for users that might be currently relying on existing config files.
Automatic merge from submit-queue
Refactor streaming code to support interop testing
Refactor exec/attach/port forward client and server code to better
support interop testing of different client and server subprotocol
versions.
Fixes#16119
os/signal.Stop() does not actually close the channel, which means
the channel is blocked. Not serious because we don't use interrupt
protection on the server, just the CLI (which can afford 1 leaked
goroutine).
Kubelet was not able to mount volumes when running inside a container and
using nsenter mounter,
NsenterMounter.IsLikelyNotMountPoint() should return ErrNotExist when the
checked directory does not exists as the regular mounted does this and
some volume plugins depend on this behavior.
Currently attach and the editor do not share the same logic for saving
and restoring the terminal, and are not suitable for nesting (when the
caller wants to create something, attach, and then delete something when
the attach is over). This commit moves the interrupt protection logic
to a util package and supports nesting interrupt handlers.
This ensures that the call to close the underlying streaming connection will execute immediately,
instead of waiting for all streams to gracefully shut down.
Due to rounding down for maxUnavailable, we may end up with rolling updates
that have zero surge and unavailable pods something that 1) is not allowed
as per validation, 2) blocks updates. If we end up in such a situation
set maxUnavailable to 1 on the theory that surge might not work due to
quota.
Had to move other things around too to avoid a weird api ->
cloudprovider dependency.
Also adding fixes per code reviews.
(This is a squash of the previously approved commits)
The message as it is framed right now does not make any sense for the
end users of our system. It might even lead to confusion. So this is
attempt to make the error message less confusing.
Update the Deployments' API types, defaulting code, conversions, helpers
and validation to use ReplicaSets instead of ReplicationControllers and
LabelSelector instead of map[string]string for selectors.
Also update the Deployment controller, registry, kubectl subcommands,
client listers package and e2e tests to use ReplicaSets and
LabelSelector for Deployments.
In order to synchronize the current state of Kubernetes's objects (e.g. pods, containers, etc.),
periodic synch loops are run. When there is a lot of objects to synchronize with,
loops increase communication traffic. At some point when all the traffic interfere cpu usage curve
hits the roof causing 100% cpu utilization.
To distribute the traffic in time, some sync loops can jitter their period in each loop
and help to flatten the curve.
This commit adds JitterUntil function with jitterFactor parameter.
JitterUntil generalizes Until which is a special case for jitterFactor being zero.
Add a mutex to guard SetUpAt() and TearDownAt() calls - they should not
run in parallel. There is a race in these calls when there are two pods
using the same volume, one of them is dying and the other one starting.
TearDownAt() checks that a volume is not needed by any pods and detaches the
volume. It does so by counting how many times is the volume mounted
(GetMountRefs() call below).
When SetUpAt() of the starting pod already attached the volume and did not mount
it yet, TearDownAt() of the dying pod will detach it - GetMountRefs() does not
count with this volume.
These two threads run in parallel:
dying pod.TearDownAt("myVolume") starting pod.SetUpAt("myVolume")
| |
| AttachDisk("myVolume")
refs, err := mount.GetMountRefs() |
Unmount("myDir") |
if refs == 1 { |
| | Mount("myVolume", "myDir")
| | |
| DetachDisk("myVolume") |
| start containers - OOPS! The volume is detached!
|
finish the pod cleanup
Also, add some logs to cinder plugin for easier debugging in the future, add
a test and update the fake mounter to know about bind mounts.
Add a recognizer that is capable of sniffing content type from data by
asking each serializer to try to decode - this is for a "universal
decoder/deserializer" which can be used by client logic.
Add codec factory, which provides the core primitives for content type
negotiation. Codec factories depend only on schemes, serializers, and
groupversion pairs.
Used like:
var pod *api.Pod
err := client.RetryOnConflict(client.DefaultBackoff, func() (err error) {
pod, err = c.Pods("mynamespace").UpdateStatus(update)
return
})
// err may be conflict
We can either fix it here or at every callsite. Every callsite is
currently using this method incorrectly.
Signed-off-by: Mike Danese <mikedanese@google.com>
I took a hard look at error output and played until I was happier. This now
prints JSON for structs in the error, rather than go's format.
Also made the error message easier to read.
Fixed tests.
Before this change we have a mish-mash of ways to pass field names around for
error generation. Sometimes string fieldnames, sometimes .Prefix(), sometimes
neither, often wrong names or not indexed when it should be.
Instead of that mess, this is part one of a couple of commits that will make it
more strongly typed and hopefully encourage correct behavior. At least you
will have to think about field names, which is better than nothing.
It turned out to be really hard to do this incrementally.
All external types that are not int64 are now marked as int32,
including
IntOrString. Prober is now int32 (43 years should be enough of an initial
probe time for anyone).
Did not change the metadata fields for now.
It's hard to manage connection buffers in a deterministic test, ensure
that the error is always consistent and the output is always a subset
of the input
Contains the following fixes for Windows users of kubectl edit:
* Defaults to notepad as the default Windows editor
* Uses CRLF line endings
* Ensures a file lock is freed
Add streaming subprotocol negotiation for exec, attach, and port
forwarding. Restore previous (buggy) exec functionality as an
unspecified/unversioned subprotocol so newer kubectl clients can work
against 1.0.x kubelets.
