Automatic merge from submit-queue
Start moving genericapiserver to staging
This moves `pkg/auth/user` to `staging/k8s.io/genericapiserver/pkg/authentication/user`. I'll open a separate pull into the upstream gengo to support using `import-boss` on vendored folders to support staging.
After we agree this is the correct approach and see everything build, I'll start moving other packages over which don't have k8s.io/kubernetes deps.
@kubernetes/sig-api-machinery-misc @lavalamp
@sttts @caesarxuchao ptal
Automatic merge from submit-queue
Fix comment and optimize code
**What this PR does / why we need it**:
Fix comment and optimize code.
Thanks.
**Special notes for your reviewer**:
**Release note**:
```release-note
```
Automatic merge from submit-queue
oidc auth-n plugin: enforce email_verified claim
This change causes the OpenID Connect authenticator to start
enforcing the 'email_verified' claim.
https://openid.net/specs/openid-connect-core-1_0.html#StandardClaims
If the OIDC authenticator uses the 'email' claim as a user's username
and the 'email_verified' is not set to `true`, reject that authentication attempt.
cc @erictune @kubernetes/sig-auth @mlbiam
```release-note
When using OIDC authentication and specifying --oidc-username-claim=email, an `"email_verified":true` claim must be returned from the identity provider.
```
Automatic merge from submit-queue (batch tested with PRs 39075, 39350, 39353)
Move pkg/api.{Context,RequestContextMapper} into pkg/genericapiserver/api/request
**Based on #39350**
Automatic merge from submit-queue
Ensure that assumed pod won't be expired until the end of binding
In case if api server is overloaded and will reply with 429 too many requests error, binding may take longer than ttl of scheduler cache for assumed pods 1199d42210/pkg/client/restclient/request.go (L787-L850)
This problem was mitigated by using this fix e4d215d508 and increased rate limit for api server. But it is possible that it will occur again.
In such cases when api server is overloaded and returns a lot of
429 (too many requests) errors - binding may take a lot of time
to succeed due to retry policy implemented in rest client.
In such events cache ttl for assumed pods wasn't big enough.
In order to minimize probability of such errors ttl for assumed pods
will be counted from the time when binding for particular pod is finished
(either with error or success)
Change-Id: Ib0122f8a76dc57c82f2c7c52497aad1bdd8be411
Automatic merge from submit-queue
Avoid unnecessary memory allocations
Low-hanging fruits in saving memory allocations. During our 5000-node kubemark runs I've see this:
ControllerManager:
- 40.17% k8s.io/kubernetes/pkg/util/system.IsMasterNode
- 19.04% k8s.io/kubernetes/pkg/controller.(*PodControllerRefManager).Classify
Scheduler:
- 42.74% k8s.io/kubernetes/plugin/pkg/scheduler/algrorithm/predicates.(*MaxPDVolumeCountChecker).filterVolumes
This PR is eliminating all of those.
Automatic merge from submit-queue
Optimize pod affinity when predicate
Optimize by returning as early as possible to avoid invoking priorityutil.PodMatchesTermsNamespaceAndSelector.
* Cache OpenID Connect clients to prevent reinitialization
* Don't retry requests in the http.RoundTripper.
* Don't rely on the server not reading POST bodies.
* Don't leak response body FDs.
* Formerly ignored any throttling requests by the server.
* Determine if the id token's expired by inspecting it.
* Similar to logic in golang.org/x/oauth2
* Synchronize around refreshing tokens and persisting the new config.
Automatic merge from submit-queue
Remove two zany unit tests.
These two tests aren't unit tests in the usual sense. We can consider switching them to run as verify checks, but I'm not convinced that they're even necessary.
They essentially work by searching their code for public functions with signatures that look like `FitPredicate`, then they shell out to grep to see that they're used somewhere in the source tree. This will never work in bazel.
Automatic merge from submit-queue
Add use case to service affinity
Also part of nits in refactoring predicates, I found the explanation of `serviceaffinity` in its comment is very hard to understand. So I added example instead here to help user/developer to digest it.
Automatic merge from submit-queue (batch tested with PRs 38154, 38502)
Rename "release_1_5" clientset to just "clientset"
We used to keep multiple releases in the main repo. Now that [client-go](https://github.com/kubernetes/client-go) does the versioning, there is no need to keep releases in the main repo. This PR renames the "release_1_5" clientset to just "clientset", clientset development will be done in this directory.
@kubernetes/sig-api-machinery @deads2k
```release-note
The main repository does not keep multiple releases of clientsets anymore. Please find previous releases at https://github.com/kubernetes/client-go
```
Automatic merge from submit-queue (batch tested with PRs 38648, 38747)
fix typo
**What this PR does / why we need it**:
fix typo.
**Release note**:
```NONE
```
Automatic merge from submit-queue (batch tested with PRs 35436, 37090, 38700)
Make iscsi pv/pvc aware of nodiskconflict feature
Being iscsi a `RWO, ROX` volume we should conflict if more than one pod is using same iscsi LUN.
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
Automatic merge from submit-queue (batch tested with PRs 38315, 38624, 38572, 38544)
scheduler: refactor main entry Run()
The kube-scheduler/app.Run() is the main entry of scheduler program.
It's enormous.
This PR tries to clean it up. Should be more modular and readable.
Automatic merge from submit-queue
Remove json serialization annotations from internal types
fixes#3933
Internal types should never be serialized, and including json serialization tags on them makes it possible to accidentally do that without realizing it.
fixes in this PR:
* types
* [x] remove json tags from internal types
* [x] fix references from serialized types to internal ObjectMeta
* generation
* [x] remove generated json codecs for internal types (they should never be used)
* kubectl
* [x] fix `apply` to operate on versioned object
* [x] fix sorting by field to operate on versioned object
* [x] fix `--record` to build annotation patch using versioned object
* hpa
* [x] fix unmarshaling to internal CustomMetricTargetList in validation
* thirdpartyresources
* [x] fix encoding API responses using internal ObjectMeta
* tests
* [x] fix tests to use versioned objects when checking encoded content
* [x] fix tests passing internal objects to generic printers
follow ups (will open tracking issues or additional PRs):
- [ ] remove json tags from internal kubeconfig types (`kubectl config set` pathfinding needs to work against external type)
- [ ] HPA should version CustomMetricTargetList serialization in annotations
- [ ] revisit how TPR resthandlers encoding objects
- [ ] audit and add tests for printer use (human-readable printer requires internal versions, generic printers require external versions)
- [ ] add static analysis tests preventing new internal types from adding tags
- [ ] add static analysis tests requiring json tags on external types (and enforcing lower-case first letter)
- [ ] add more tests for `kubectl get` exercising known and unknown types with all output options
This change causes the OpenID Connect authenticator to start
enforcing the 'email_verified' claim.
https://openid.net/specs/openid-connect-core-1_0.html#StandardClaims
If the OIDC authenticator uses the 'email' claim as a user's password
and the 'email_verified' holds the value false, reject that
authentication attempt.
If 'email_verified' is true or not present, continue as before.
Automatic merge from submit-queue (batch tested with PRs 38377, 36365, 36648, 37691, 38339)
Do not create selector and namespaces in a loop where possible
With 1000 nodes and 5000 pods (5 pods per node) with anti-affinity a lot of CPU wasted on creating LabelSelector and sets.String (map).
With this change we are able to deploy that number of pods in ~25 minutes. Without - it takes 30 minutes to deploy 500 pods with anti-affinity configured.
failureDomains are only used for PreferredDuringScheduling pod
anti-affinity, which is ignored by PodAffinityChecker.
This unnecessary requirement was making it hard to move
PodAffinityChecker to GeneralPredicates because that would require
passing --failure-domains to both kubelet and kube-controller-manager.
Automatic merge from submit-queue (batch tested with PRs 35101, 38215, 38092)
auth duplicate detect
I think we should not allow people set duplicate tokens in token file or set duplicate usernames in password file. because the default action overwriting the old data may let people misunderstand.
Automatic merge from submit-queue
add default label to rbac bootstrap policy
allow people to retrieve information of bootstrap policy by label :
`kubectl get clusterroles -l key=value`
`kubectl get clusterrolebindings -l key=value`
Automatic merge from submit-queue (batch tested with PRs 38076, 38137, 36882, 37634, 37558)
[scheduler] Use V(10) for anything which may be O(N*P) logging
Fixes#37014
This PR makes sure that logging statements which are capable of being called on a perNode / perPod basis (i.e. non essential ones that will just clog up logs at large scale) are at V(10) level.
I dreamt of a levenstein filter that built a weak map of word frequencies and alerted once log throughput increased w/o varying information content.... but then I woke up and realized this is probably all we really need for now :)
Automatic merge from submit-queue (batch tested with PRs 38111, 38121)
remove rbac super user
Cleaning up cruft and duplicated capabilities as we transition from RBAC alpha to beta. In 1.5, we added a secured loopback connection based on the `system:masters` group name. `system:masters` have full power in the API, so the RBAC super user is superfluous.
The flag will stay in place so that the process can still launch, but it will be disconnected.
@kubernetes/sig-auth
Automatic merge from submit-queue (batch tested with PRs 36816, 37534)
Move pkg/api/unversioned to pkg/apis/meta/v1
This moves code from using pkg/api/unversioned to pkg/apis/meta/v1 with the `metav1` local package name.
Built on top of #37532 (the first three commits related to ExportOptions)
Part of #37530
Automatic merge from submit-queue
plumb in front proxy group header
Builds on https://github.com/kubernetes/kubernetes/pull/36662 and https://github.com/kubernetes/kubernetes/pull/36774, so only the last commit is unique.
This completes the plumbing for front proxy header information and makes it possible to add just the front proxy header authenticator.
WIP because I'm going to assess it in use downstream.
Automatic merge from submit-queue (batch tested with PRs 35300, 36709, 37643, 37813, 37697)
add rbac action to subjects type
This adds the ability to go from an authorization action to the list subjects who have the power to perform the action. This will be used to either back an RBAC specific endpoint or generic authorization endpoint. Because of the way authorization works today, the set of subjects returned will always be a subset of those with access since any authorizer can say yes.
@kubernetes/sig-auth
Automatic merge from submit-queue (batch tested with PRs 37945, 37498, 37391, 37209, 37169)
add controller roles
Upstream controller roles that have downstream.
@sttts this is a start at roles for controllers. I've made names match for now, but they could use some love in both the controller manager and here. I'd recommend using this as a starting point.
Automatic merge from submit-queue (batch tested with PRs 36263, 36755, 37357, 37222, 37524)
Add flag to enable contention profiling in scheduler.
```release-note
Add flag to enable contention profiling in scheduler.
```
Automatic merge from submit-queue
auth delegation role
Add a bootstrap role for authentication and authorization delegation. Useful for extension API servers.
@kubernetes/sig-auth
Automatic merge from submit-queue
Revert "Avoid hard-coding list of Node Conditions"
* we don't know how other API consumers are using node conditions (there was no prior expectation that the scheduler would block on custom conditions)
* not all conditions map directly to schedulability (e.g. `MemoryPressure`/`DiskPressure`)
* not all conditions use True to mean "unschedulable" (e.g. `Ready`)
This reverts commit 511b2ecaa8 to avoid breaking existing API users and to avoid constraining future uses of the node conditions API
Automatic merge from submit-queue
Add authz to psp admission
Add authz integration to PSP admission to enable granting access to use specific PSPs on a per-user and per-service account basis. This allows an administrator to use multiple policies in a cluster that grant different levels of access for different types of users.
Builds on https://github.com/kubernetes/kubernetes/pull/32555. Second commit adds authz check to matching policy function in psp admission.
@deads2k @sttts @timstclair
Automatic merge from submit-queue
specify custom ca file to verify the keystone server
<!-- Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, read our contributor guidelines https://github.com/kubernetes/kubernetes/blob/master/CONTRIBUTING.md and developer guide https://github.com/kubernetes/kubernetes/blob/master/docs/devel/development.md
2. If you want *faster* PR reviews, read how: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/faster_reviews.md
3. Follow the instructions for writing a release note: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/pull-requests.md#release-notes
-->
**What this PR does / why we need it**:
Sometimes the keystone server's certificate is self-signed, mainly used for internal development, testing and etc.
For this kind of ca, we need a way to verify the keystone server.
Otherwise, below error will occur.
> x509: certificate signed by unknown authority
This patch provide a way to pass in a ca file to verify the keystone server when starting `kube-apiserver`.
**Which issue this PR fixes** : fixes#22695, #24984
**Special notes for your reviewer**:
**Release note**:
<!-- Steps to write your release note:
1. Use the release-note-* labels to set the release note state (if you have access)
2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write `NONE`.
-->
``` release-note
```
Automatic merge from submit-queue
'Max' and 'MIn' don't seem to used anywhere, so I would suggest removing them
Signed-off-by: Yanqiang Miao miao.yanqiang@zte.com.cn
Automatic merge from submit-queue
Rename ScheduledJobs to CronJobs
I went with @smarterclayton idea of registering named types in schema. This way we can support both the new (CronJobs) and old (ScheduledJobs) resource name. Fixes#32150.
fyi @erictune @caesarxuchao @janetkuo
Not ready yet, but getting close there...
**Release note**:
```release-note
Rename ScheduledJobs to CronJobs.
```
Automatic merge from submit-queue
We only report diskpressure to users, and no longer report inodepressure
See #36180 for more information on why #33218 was reverted.
Automatic merge from submit-queue
Add cmd support to gcp auth provider plugin
**What this PR does / why we need it**:
Adds ability for gcp auth provider plugin to get access token by shelling out to an external command. We need this because for GKE, kubectl should be using gcloud credentials. It currently uses google application default credentials, which causes confusion if user has configured both with different permissions (previously the two were almost always identical).
**Which issue this PR fixes**:
Addresses #35530 with gcp-only solution, as generic cmd plugin was deemed not useful for other providers.
**Special notes for your reviewer**:
Configuration options are to support whatever future command gcloud provides for printing access token of active user. Also works with existing command (`gcloud auth print-access-token`)
```release-note
```
We are more liberal in what we accept as a volume id in k8s, and indeed
we ourselves generate names that look like `aws://<zone>/<id>` for
dynamic volumes.
This volume id (hereafter a KubernetesVolumeID) cannot directly be
compared to an AWS volume ID (hereafter an awsVolumeID).
We introduce types for each, to prevent accidental comparison or
confusion.
Issue #35746
Automatic merge from submit-queue
split scheduler priorities into separate files
In the current state it's really hard to find a thing one is looking for, if he doesn't know already know where to look. cc @davidopp
Automatic merge from submit-queue
Add tooling to generate listers
Add lister-gen tool to auto-generate listers. So far this PR only demonstrates replacing the manually-written `StoreToLimitRangeLister` with the generated `LimitRangeLister`, as it's a small and easy swap.
cc @deads2k @liggitt @sttts @nikhiljindal @lavalamp @smarterclayton @derekwaynecarr @kubernetes/sig-api-machinery @kubernetes/rh-cluster-infra
Automatic merge from submit-queue
Update PodAntiAffinity to ignore calls to subresources
@smarterclayton I hit this when I was trying to evict a pod, apparently k8s does not have this particular admission plugin on by default. ptal
@mml @davidopp fyi
Automatic merge from submit-queue
allow authentication through a front-proxy
This allows a front proxy to set a request header and have that be a valid `user.Info` in the authentication chain. To secure this power, a client certificate may be used to confirm the identity of the front proxy
@kubernetes/sig-auth fyi
@erictune per-request
@liggitt you wrote the openshift one, ptal.
Automatic merge from submit-queue
Simplify negotiation in server in preparation for multi version support
This is a pre-factor for #33900 to simplify runtime.NegotiatedSerializer, tighten up a few abstractions that may break when clients can request different client versions, and pave the way for better negotiation.
View this as pure simplification.
Automatic merge from submit-queue
[PHASE 1] Opaque integer resource accounting.
## [PHASE 1] Opaque integer resource accounting.
This change provides a simple way to advertise some amount of arbitrary countable resource for a node in a Kubernetes cluster. Users can consume these resources by including them in pod specs, and the scheduler takes them into account when placing pods on nodes. See the example at the bottom of the PR description for more info.
Summary of changes:
- Defines opaque integer resources as any resource with prefix `pod.alpha.kubernetes.io/opaque-int-resource-`.
- Prevent kubelet from overwriting capacity.
- Handle opaque resources in scheduler.
- Validate integer-ness of opaque int quantities in API server.
- Tests for above.
Feature issue: https://github.com/kubernetes/features/issues/76
Design: http://goo.gl/IoKYP1
Issues:
kubernetes/kubernetes#28312kubernetes/kubernetes#19082
Related:
kubernetes/kubernetes#19080
CC @davidopp @timothysc @balajismaniam
**Release note**:
<!-- Steps to write your release note:
1. Use the release-note-* labels to set the release note state (if you have access)
2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write `NONE`.
-->
```release-note
Added support for accounting opaque integer resources.
Allows cluster operators to advertise new node-level resources that would be
otherwise unknown to Kubernetes. Users can consume these resources in pod
specs just like CPU and memory. The scheduler takes care of the resource
accounting so that no more than the available amount is simultaneously
allocated to pods.
```
## Usage example
```sh
$ echo '[{"op": "add", "path": "pod.alpha.kubernetes.io~1opaque-int-resource-bananas", "value": "555"}]' | \
> http PATCH http://localhost:8080/api/v1/nodes/localhost.localdomain/status \
> Content-Type:application/json-patch+json
```
```http
HTTP/1.1 200 OK
Content-Type: application/json
Date: Thu, 11 Aug 2016 16:44:55 GMT
Transfer-Encoding: chunked
{
"apiVersion": "v1",
"kind": "Node",
"metadata": {
"annotations": {
"volumes.kubernetes.io/controller-managed-attach-detach": "true"
},
"creationTimestamp": "2016-07-12T04:07:43Z",
"labels": {
"beta.kubernetes.io/arch": "amd64",
"beta.kubernetes.io/os": "linux",
"kubernetes.io/hostname": "localhost.localdomain"
},
"name": "localhost.localdomain",
"resourceVersion": "12837",
"selfLink": "/api/v1/nodes/localhost.localdomain/status",
"uid": "2ee9ea1c-47e6-11e6-9fb4-525400659b2e"
},
"spec": {
"externalID": "localhost.localdomain"
},
"status": {
"addresses": [
{
"address": "10.0.2.15",
"type": "LegacyHostIP"
},
{
"address": "10.0.2.15",
"type": "InternalIP"
}
],
"allocatable": {
"alpha.kubernetes.io/nvidia-gpu": "0",
"cpu": "2",
"memory": "8175808Ki",
"pods": "110"
},
"capacity": {
"alpha.kubernetes.io/nvidia-gpu": "0",
"pod.alpha.kubernetes.io/opaque-int-resource-bananas": "555",
"cpu": "2",
"memory": "8175808Ki",
"pods": "110"
},
"conditions": [
{
"lastHeartbeatTime": "2016-08-11T16:44:47Z",
"lastTransitionTime": "2016-07-12T04:07:43Z",
"message": "kubelet has sufficient disk space available",
"reason": "KubeletHasSufficientDisk",
"status": "False",
"type": "OutOfDisk"
},
{
"lastHeartbeatTime": "2016-08-11T16:44:47Z",
"lastTransitionTime": "2016-07-12T04:07:43Z",
"message": "kubelet has sufficient memory available",
"reason": "KubeletHasSufficientMemory",
"status": "False",
"type": "MemoryPressure"
},
{
"lastHeartbeatTime": "2016-08-11T16:44:47Z",
"lastTransitionTime": "2016-08-10T06:27:11Z",
"message": "kubelet is posting ready status",
"reason": "KubeletReady",
"status": "True",
"type": "Ready"
},
{
"lastHeartbeatTime": "2016-08-11T16:44:47Z",
"lastTransitionTime": "2016-08-10T06:27:01Z",
"message": "kubelet has no disk pressure",
"reason": "KubeletHasNoDiskPressure",
"status": "False",
"type": "DiskPressure"
}
],
"daemonEndpoints": {
"kubeletEndpoint": {
"Port": 10250
}
},
"images": [],
"nodeInfo": {
"architecture": "amd64",
"bootID": "1f7e95ca-a4c2-490e-8ca2-6621ae1eb5f0",
"containerRuntimeVersion": "docker://1.10.3",
"kernelVersion": "4.5.7-202.fc23.x86_64",
"kubeProxyVersion": "v1.3.0-alpha.4.4285+7e4b86c96110d3-dirty",
"kubeletVersion": "v1.3.0-alpha.4.4285+7e4b86c96110d3-dirty",
"machineID": "cac4063395254bc89d06af5d05322453",
"operatingSystem": "linux",
"osImage": "Fedora 23 (Cloud Edition)",
"systemUUID": "D6EE0782-5DEB-4465-B35D-E54190C5EE96"
}
}
}
```
After patching, the kubelet's next sync fills in allocatable:
```
$ kubectl get node localhost.localdomain -o json | jq .status.allocatable
```
```json
{
"alpha.kubernetes.io/nvidia-gpu": "0",
"pod.alpha.kubernetes.io/opaque-int-resource-bananas": "555",
"cpu": "2",
"memory": "8175808Ki",
"pods": "110"
}
```
Create two pods, one that needs a single banana and another that needs a truck load:
```
$ kubectl create -f chimp.yaml
$ kubectl create -f superchimp.yaml
```
Inspect the scheduler result and pod status:
```
$ kubectl describe pods chimp
Name: chimp
Namespace: default
Node: localhost.localdomain/10.0.2.15
Start Time: Thu, 11 Aug 2016 19:58:46 +0000
Labels: <none>
Status: Running
IP: 172.17.0.2
Controllers: <none>
Containers:
nginx:
Container ID: docker://46ff268f2f9217c59cc49f97cc4f0f085d5ac0e251f508cc08938601117c0cec
Image: nginx:1.10
Image ID: docker://sha256:82e97a2b0390a20107ab1310dea17f539ff6034438099384998fd91fc540b128
Port: 80/TCP
Limits:
cpu: 500m
memory: 64Mi
pod.alpha.kubernetes.io/opaque-int-resource-bananas: 3
Requests:
cpu: 250m
memory: 32Mi
pod.alpha.kubernetes.io/opaque-int-resource-bananas: 1
State: Running
Started: Thu, 11 Aug 2016 19:58:51 +0000
Ready: True
Restart Count: 0
Volume Mounts: <none>
Environment Variables: <none>
Conditions:
Type Status
Initialized True
Ready True
PodScheduled True
No volumes.
QoS Class: Burstable
Events:
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
9m 9m 1 {default-scheduler } Normal Scheduled Successfully assigned chimp to localhost.localdomain
9m 9m 2 {kubelet localhost.localdomain} Warning MissingClusterDNS kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to DNSDefault policy.
9m 9m 1 {kubelet localhost.localdomain} spec.containers{nginx} Normal Pulled Container image "nginx:1.10" already present on machine
9m 9m 1 {kubelet localhost.localdomain} spec.containers{nginx} Normal Created Created container with docker id 46ff268f2f92
9m 9m 1 {kubelet localhost.localdomain} spec.containers{nginx} Normal Started Started container with docker id 46ff268f2f92
```
```
$ kubectl describe pods superchimp
Name: superchimp
Namespace: default
Node: /
Labels: <none>
Status: Pending
IP:
Controllers: <none>
Containers:
nginx:
Image: nginx:1.10
Port: 80/TCP
Requests:
cpu: 250m
memory: 32Mi
pod.alpha.kubernetes.io/opaque-int-resource-bananas: 10Ki
Volume Mounts: <none>
Environment Variables: <none>
Conditions:
Type Status
PodScheduled False
No volumes.
QoS Class: Burstable
Events:
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
3m 1s 15 {default-scheduler } Warning FailedScheduling pod (superchimp) failed to fit in any node
fit failure on node (localhost.localdomain): Insufficient pod.alpha.kubernetes.io/opaque-int-resource-bananas
```
Automatic merge from submit-queue
Add sync state loop in master's volume reconciler
At master volume reconciler, the information about which volumes are
attached to nodes is cached in actual state of world. However, this
information might be out of date in case that node is terminated (volume
is detached automatically). In this situation, reconciler assume volume
is still attached and will not issue attach operation when node comes
back. Pods created on those nodes will fail to mount.
This PR adds the logic to periodically sync up the truth for attached
volumes kept in
the actual state cache. If the volume is no longer attached to the node,
the actual state will be updated to reflect the truth. In turn,
reconciler will take actions if needed.
To avoid issuing many concurrent operations on cloud provider, this PR
tries to add batch operation to check whether a list of volumes are
attached to the node instead of one request per volume.
- Prevents kubelet from overwriting capacity during sync.
- Handles opaque integer resources in the scheduler.
- Adds scheduler predicate tests for opaque resources.
- Validates opaque int resources:
- Ensures supplied opaque int quantities in node capacity,
node allocatable, pod request and pod limit are integers.
- Adds tests for new validation logic (node update and pod spec).
- Added e2e tests for opaque integer resources.
At master volume reconciler, the information about which volumes are
attached to nodes is cached in actual state of world. However, this
information might be out of date in case that node is terminated (volume
is detached automatically). In this situation, reconciler assume volume
is still attached and will not issue attach operation when node comes
back. Pods created on those nodes will fail to mount.
This PR adds the logic to periodically sync up the truth for attached volumes kept in the actual state cache. If the volume is no longer attached to the node, the actual state will be updated to reflect the truth. In turn, reconciler will take actions if needed.
To avoid issuing many concurrent operations on cloud provider, this PR
tries to add batch operation to check whether a list of volumes are
attached to the node instead of one request per volume.
More details are explained in PR #33760
Alter how runtime.SerializeInfo is represented to simplify negotiation
and reduce the need to allocate during negotiation. Simplify the dynamic
client's logic around negotiating type. Add more tests for media type
handling where necessary.
Automatic merge from submit-queue
Make "Unschedulable" reason a constant in api
String "Unschedulable" is used in couple places in K8S:
* scheduler
* federation replicaset and deployment controllers
* cluster autoscaler
* rescheduler
This PR makes the string a part of API so it not changed.
cc: @davidopp @fgrzadkowski @wojtek-t
Automatic merge from submit-queue
Create restclient interface
Refactoring of code to allow replace *restclient.RESTClient with any RESTClient implementation that implements restclient.RESTClientInterface interface.
Automatic merge from submit-queue
Predicate cacheing and cleanup
Fix to #31795
First pass @ cleanup and caching of the CheckServiceAffinity function.
The cleanup IMO is necessary because the logic around the pod listing and the use of the "implicit selector" (which is reverse engineered to enable the homogenous pod groups).
Should still pass the E2Es.
@timothysc @wojtek-t
Comments addressed, Make emptyMetadataProducer a func to avoid casting,
FakeSvcLister: remove error return for len(svc)=0. New test for
predicatePrecomp to make method semantics explictly enforced when meta
is missing. Precompute wrapper.
Automatic merge from submit-queue
PVC informer lister supports listing
This will be used in follow-on PRs for quota evaluation backed by informers for pvcs.
/cc @deads2k @eparis @kubernetes/sig-storage
Automatic merge from submit-queue
Adding default StorageClass annotation printout for resource_printer and describer and some refactoring
adding ISDEFAULT for _kubectl get storageclass_ output
```
[root@screeley-sc1 gce]# kubectl get storageclass
NAME TYPE ISDEFAULT
another-class kubernetes.io/gce-pd NO
generic1-slow kubernetes.io/gce-pd YES
generic2-fast kubernetes.io/gce-pd YES
```
```release-note
Add ISDEFAULT to kubectl get storageClass output
```
@kubernetes/sig-storage
Most normal codec use should perform defaulting. DirectCodecs should not
perform defaulting. Update the defaulting_test to fuzz the list of known
defaulters. Use the new versioning.NewDefaultingCodec() method.
Automatic merge from submit-queue
Add PSP support for seccomp profiles
Seccomp support for PSP. There are still a couple of TODOs that need to be fixed but this is passing tests.
One thing of note, since seccomp is all being stored in annotations right now it breaks some of the assumptions we've stated for the provider in terms of mutating the passed in pod. I've put big warning comments around the pieces that do that to make sure it's clear and covered the rollback in admission if the policy fails to validate.
@sttts @pmorie @erictune @smarterclayton @liggitt
Automatic merge from submit-queue
Refactor scheduler to enable switch on equivalence cache
Part 2 of #30844
Refactoring to enable easier switch on of equivalence cache.
Automatic merge from submit-queue
add ownerref permission checks
Adds an admission plugin that ensures that anyone adding an `ownerReference` to a resource has delete rights on the resource they're setting up a delete for.
@caesarxuchao example admission plugin that tests for ownerReference diffs and uses an authorizer to drive the decision.
@liggitt @ncdc we've talked about this before
Automatic merge from submit-queue
Fix typos and englishify plugin/pkg
**What this PR does / why we need it**: Just typos
**Which issue this PR fixes**: `None`
**Special notes for your reviewer**: Just typos
**Release note**: `NONE`
Automatic merge from submit-queue
Add support for admission controller based on namespace node selectors.
This work is to upstream openshift's project node selectors based admission controller.
Fixes https://github.com/kubernetes/kubernetes/issues/17151
Automatic merge from submit-queue
Run rbac authorizer from cache
RBAC authorization can be run very effectively out of a cache. The cache is a normal reflector backed cache (shared informer).
I've split this into three parts:
1. slim down the authorizer interfaces
1. boilerplate for adding rbac shared informers and associated listers which conform to the new interfaces
1. wiring
@liggitt @ericchiang @kubernetes/sig-auth
Automatic merge from submit-queue
Remove scheduler flags that were marked as deprecated 2+ releases ago.
Starting to spin back into scheduler testing and performance evaluation, and I noticed some leftover cleanup work from 2+ releases ago.
**Release note**: Removed unused bind-pods-qps, and bind-pods-burst flags from the scheduler.
Automatic merge from submit-queue
Minor cleanup to scheduler.
Part 1 of pr for scheduler modularization specifically supporting CheckServiceAffinity improvements #33763.
These are some innoccous changes which help separate the predicate injection machinery from the cleanup parts of CheckServiceAffinity
@timothysc @wojtek-t
- changes to metadata.go are for the follow on pr
- changes to prio/pred stuff is all passing unit tests
hoping to grease this through and then the interesting part is in the follow on pr.
Automatic merge from submit-queue
WantsAuthorizer admission plugin support
The next step of PSP admission is to be able to limit the PSPs used based on user information. To do this the admission plugin would need to make authz checks for the `user.Info` in the request. This code allows a plugin to request the injection of an authorizer to allow it to make the authz checks.
Note: this could be done with a SAR, however since admission is running in the api server using the SAR would incur an extra hop vs using the authorizer directly.
@deads2k @derekwaynecarr
Automatic merge from submit-queue
Add PVC storage to LimitRange
This PR adds the ability to add a LimitRange to a namespace that enforces min/max on `pvc.Spec.Resources.Requests["storage"]`.
@derekwaynecarr @abhgupta @kubernetes/sig-storage
Examples forthcoming.
```release-note
pvc.Spec.Resources.Requests min and max can be enforced with a LimitRange of type "PersistentVolumeClaim" in the namespace
```
Automatic merge from submit-queue
Allow webhook authorizer to use SubjectAccessReviewInterface
Refactors the authorization webhook to be able to be fed a kubeconfig file or a SubjectAccessReviewsInterface
Added tests to exercise retry logic, and ensure the correct serialized version is sent to the remote webhook (I also made sure the new tests passed on the current webhook impl in master)
c.f. https://github.com/kubernetes/kubernetes/pull/32547
c.f. https://github.com/kubernetes/kubernetes/pull/32518
Automatic merge from submit-queue
[Part 1] Implementation of equivalence pod
Part 1 of #30844
This PR:
- Refactored `predicate.go`, so that `GetResourceRequest` can be used in other places to `GetEquivalencePod`.
- Implement a `equivalence_cache.go` to deal with all information we need to calculate an equivalent pod.
- Define and register a `RegisterGetEquivalencePodFunction `.
Work in next PR:
- The update of `equivalence_cache.go`
- Unit test
- Integration/e2e test
I think we can begin from the `equivalence_cache.go`? Thanks. cc @wojtek-t @davidopp
If I missed any other necessary part, please point it out.
Automatic merge from submit-queue
Add kubelet awareness to taint tolerant match caculator.
Add kubelet awareness to taint tolerant match caculator.
Ref: #25320
This is required by `TaintEffectNoScheduleNoAdmit` & `TaintEffectNoScheduleNoAdmitNoExecute `, so that node will know if it should expect the taint&tolerant
Automatic merge from submit-queue
Turn down hootloop logs in priorities.
Excessive spam is output once we near cluster capacity, sometimes a panic is triggered but that data is clipped in the logs see #33935 for more details.
Automatic merge from submit-queue
update deployment and replicaset listers
Updates the deployment lister to avoid copies and updates the deployment controller to use shared informers.
Pushing WIP to see which tests are broken.
Automatic merge from submit-queue
decouple workqueue metrics from prometheus
<!-- Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, read our contributor guidelines https://github.com/kubernetes/kubernetes/blob/master/CONTRIBUTING.md and developer guide https://github.com/kubernetes/kubernetes/blob/master/docs/devel/development.md
2. If you want *faster* PR reviews, read how: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/faster_reviews.md
3. Follow the instructions for writing a release note: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/pull-requests.md#release-notes
-->
**What this PR does / why we need it**:
We want to include the workqueue in client-go, but do not want to having to import Prometheus. This PR decouples the workqueue from prometheus.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
Partially address https://github.com/kubernetes/kubernetes/issues/33497
User requested for `workqueue` in client-go: https://github.com/kubernetes/client-go/issues/4#issuecomment-249444848
**Special notes for your reviewer**:
**Release note**:
<!-- Steps to write your release note:
1. Use the release-note-* labels to set the release note state (if you have access)
2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write `NONE`.
-->
```release-note
The implicit registration of Prometheus metrics for workqueue has been removed, and a plug-able interface was added. If you were using workqueue in your own binaries and want these metrics, add the following to your imports in the main package: "k8s.io/pkg/util/workqueue/prometheus".
```
Automatic merge from submit-queue
scheduler: initialize podsWithAffinity
Without initializing podsWithAffinity, scheduler panics when deleting
a pod from a node that has no pods with affinity ever scheduled to.
Fix#33772
Automatic merge from submit-queue
pass loopback config to poststart hooks
Updates post start hooks to take a clientconfig with the new loopback credentials for bootstrapping.
@ericchiang This is a little bit of plumbing, but mainly auth I think.
Automatic merge from submit-queue
Allow anonymous API server access, decorate authenticated users with system:authenticated group
When writing authorization policy, it is often necessary to allow certain actions to any authenticated user. For example, creating a service or configmap, and granting read access to all users
It is also frequently necessary to allow actions to any unauthenticated user. For example, fetching discovery APIs might be part of an authentication process, and therefore need to be able to be read without access to authentication credentials.
This PR:
* Adds an option to allow anonymous requests to the secured API port. If enabled, requests to the secure port that are not rejected by other configured authentication methods are treated as anonymous requests, and given a username of `system:anonymous` and a group of `system:unauthenticated`. Note: this should only be used with an `--authorization-mode` other than `AlwaysAllow`
* Decorates user.Info returned from configured authenticators with the group `system:authenticated`.
This is related to defining a default set of roles and bindings for RBAC (https://github.com/kubernetes/features/issues/2). The bootstrap policy should allow all users (anonymous or authenticated) to request the discovery APIs.
```release-note
kube-apiserver learned the '--anonymous-auth' flag, which defaults to true. When enabled, requests to the secure port that are not rejected by other configured authentication methods are treated as anonymous requests, and given a username of 'system:anonymous' and a group of 'system:unauthenticated'.
Authenticated users are decorated with a 'system:authenticated' group.
NOTE: anonymous access is enabled by default. If you rely on authentication alone to authorize access, change to use an authorization mode other than AlwaysAllow, or or set '--anonymous-auth=false'.
```
c.f. https://github.com/kubernetes/kubernetes/issues/29177#issuecomment-244191596
We had another bug where we confused the hostname with the NodeName.
To avoid this happening again, and to make the code more
self-documenting, we use types.NodeName (a typedef alias for string)
whenever we are referring to the Node.Name.
A tedious but mechanical commit therefore, to change all uses of the
node name to use types.NodeName
Also clean up some of the (many) places where the NodeName is referred
to as a hostname (not true on AWS), or an instanceID (not true on GCE),
etc.
Automatic merge from submit-queue
Copy auth plugin to client-go repo
client-go doesn't copy the [auth plugin](https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/client/auth/plugins.go). This causes user cannot access cluster run by GKE. User will see error "No Auth Provider found for name gcp".
This PR fixes this issue. It's marked as WIP because I'll need to rebase after #32906 gets merged. Also, the fix needs to be cherry-picked into 1.4 branch to update client-go/1.4.
Automatic merge from submit-queue
add system:discovery role
Adds a role for version negotiation from `kubectl`.
@mlbiam ptal. If it works for you, I think we can merge it.
@kubernetes/sig-auth fyi
Automatic merge from submit-queue
Fix cache expiration check
The check for whether an entry in the `forceLiveLookup` cache had expired was backwards. Fixed the logic and added tests
rename plugin/pkg/client/auth/plugins.go package name to auth
add the plugin import line in client-gen
update import_known_versions for release_1_5 clientset
change copy.sh
Automatic merge from submit-queue
simplify RC listers
Make the RC and SVC listers use the common list functions that more closely match client APIs, are consistent with other listers, and avoid unnecessary copies.
Automatic merge from submit-queue
oidc authenticator: allow string value as groups claim
Allow the group claim to be a single string instead of an array of
strings. This means the following claim
{
"role": "admin"
}
Will be mapped to the groups
["admin"]
cc @kubernetes/sig-auth @mlbiam
closes#33290
Automatic merge from submit-queue
Staging 1.5 client
Created the 1.5 folder and remove the 1.4 folder in the staging area in the master branch.
Content of kubernetes/client-go/1.4 will be pulled from the kubernetes/kubernetes 1.4 branch (https://github.com/kubernetes/contrib/pull/1719)
Allow the group claim to be a single string instead of an array of
strings. This means the following claim
{
"role": "admin"
}
Will be mapped to the groups
["admin"]
Automatic merge from submit-queue
Refactor cert utils into one pkg, add funcs from bootkube for kubeadm to use
**What this PR does / why we need it**:
We have ended-up with rather incomplete and fragmented collection of utils for handling certificates. It may be worse to consider using `cfssl` for doing all of these things, but for now there is some functionality that we need in `kubeadm` that we can borrow from bootkube. It makes sense to move the utils from bookube into core, as discussed in #31221.
**Special notes for your reviewer**: I've taken the opportunity to review names of existing funcs and tried to make some improvements in that area (with help from @peterbourgon).
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue
Allow secure access to apiserver from Admission Controllers
* Allow options.InsecurePort to be set to 0 to switch off insecure access
* In NewSelfClient, Set the TLSClientConfig to the cert and key files
if InsecurePort is switched off
* Mint a bearer token that allows the client(s) created in NewSelfClient
to talk to the api server
* Add a new authenticator that checks for this specific bearer token
Fixes#13598
Automatic merge from submit-queue
simplify RC and SVC listers
Make the RC and SVC listers use the common list functions that more closely match client APIs, are consistent with other listers, and avoid unnecessary copies.
Automatic merge from submit-queue
Fix possible panic in PodAffinityChecker
In `PodAffinityChecker.getMatchingAntiAffinityTerms()` `affinity.PodAntiAffinity` can be `nil` in this place. This problem occurs e.g. when `nil` is passed as `meta` to `PodAffinityChecker.InterPodAffinityMatches()`.
Stumbled upon it while working on #31136 (someone PTAL at that PR too, I've submitted it a month ago and seemingly no one noticed it), kube-controller-manager was crashing there.
* Allow options.InsecurePort to be set to 0 to switch off insecure access
* In NewSelfClient, Set the TLSClientConfig to the cert and key files
if InsecurePort is switched off
* Mint a bearer token that allows the client(s) created in NewSelfClient
to talk to the api server
* Add a new authenticator that checks for this specific bearer token
Fixes#13598
Automatic merge from submit-queue
Fix minor nits in test cases
Found a group of nits when doing #30844, fixed them in a this PR since 30844 requires a long time to review.
Automatic merge from submit-queue
Exit scheduler retry loop correctly
The error was being eaten and shadowed, which means we would never exit
this loop. This might lead to a goroutine in the scheduler being used
forever without exiting at maximum backoff.
Switched to use the real client to make later refactors easier.
@wojtek-t this may lead to scheduler informer exhaustion - not that familiar with retries
Automatic merge from submit-queue
plugin/pkg: add ericchiang to owners for OpenID Connect plugins
On the CoreOS side I'm taking over these components.
@erictune has suggested `pkg/registry/{cluster}role{binding}s/OWNERS` as well but I'd feel more comfortable as a reviewer for those specific parts of RBAC for a couple more cycles since @liggitt and @deads2k have had way more experience in the registry code.
Beyond that the only existing OWNERS files for auth are `pkg/auth/OWNERS` and `plugin/pkg/auth/OWNERS` both of which include @liggitt and @erictune. There's also nothing in the `plugin/pkg/client` path. I'm a little unclear on the implications for future PRs that are assigned to me (e.g. webhook or RBAC reviews).
cc @kubernetes/sig-auth
The error was being eaten and shadowed, which means we would never exit
this loop. This might lead to a goroutine in the scheduler being used
forever without exiting at maximum backoff.
Switched to use the real client to make later refactors easier.
Bump version of golang.org/x/oauth2
Vendor google.golang.org/cloud/
Vendor google.golang.org/api/
Vendor cloud.google.com/go/compute/
Replace google.golang.org/cloud with cloud.google.com/go/
Fixes#30069
Automatic merge from submit-queue
Allow webhook authenticator to use TokenReviewsInterface
Refactors the authentication webhook to be able to be fed a kubeconfig file or a TokenReviewsInterface
Fixes a bug with webhooks not retrying on 429/500 errors
Related to https://github.com/kubernetes/kubernetes/pull/32518#discussion_r78450170
Automatic merge from submit-queue
Ensure that we are closing files.
**What this PR does / why we need it**: In several places we are leaking file descriptors. This could be problematic on systems with low ulimits for them.
**Release note**:
```release-note
```
Automatic merge from submit-queue
Change rbac roleref to reflect the information we want
@liggitt @ericchiang This is a version of https://github.com/kubernetes/kubernetes/pull/31359 which updates the `RoleRef` to be (I think) the type that we want, with a group, resource, and name.
This is **not** backwards compatible with any existing data. I'm ok with doing this since rbac was considered alpha, but its something to consider.
If we want this instead, I'll close the previous pull (or update it with this content).
Automatic merge from submit-queue
Update scheduler config file compatibility tests
**What this PR does / why we need it**:
Added missing compatibility tests for scheduler config file options.
**Which issue this PR fixes**
fixes#30099
**Special notes for your reviewer**:
I came up with the options based on the contents of default.go in each branch.
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue
make deep copy of quota objects before mutations
The code currently makes shallow copies which ensures that we aren't accidentally reslicing anything in weird ways, but the usage maps are pointers, so they end up being shared.
This makes a couple copies when we know we're going to mutate to avoid changing shared maps.
Automatic merge from submit-queue
ClientConfig should not default to http://localhost:8080
This changes clientcmd to skip the default cluster, but preserves the
behavior in kubectl. This prevents the possibility of an administrator
misconfiguration in kubelet or other server component from allowing a
third party who can bind to 8080 on that host from potentially
impersonating an API server and gaining root access.
@mikedanese @deads2k this removes the defaulting of http://localhost:8080 for server from everything except kubectl.
```release-note
Kubernetes server components using `kubeconfig` files no longer default to `http://localhost:8080`. Administrators must specify a server value in their kubeconfig files.
```
This changes clientcmd to skip the default cluster, but preserves the
behavior in kubectl. This prevents the possibility of an administrator
misconfiguration in kubelet or other server component from allowing a
third party who can bind to 8080 on that host from potentially
impersonating an API server and gaining root access.
Automatic merge from submit-queue
Add sysctl support
Implementation of proposal https://github.com/kubernetes/kubernetes/pull/26057, feature https://github.com/kubernetes/features/issues/34
TODO:
- [x] change types.go
- [x] implement docker and rkt support
- [x] add e2e tests
- [x] decide whether we want apiserver validation
- ~~[ ] add documentation~~: api docs exist. Existing PodSecurityContext docs is very light and links back to the api docs anyway: 6684555ed9/docs/user-guide/security-context.md
- [x] change PodSecurityPolicy in types.go
- [x] write admission controller support for PodSecurityPolicy
- [x] write e2e test for PodSecurityPolicy
- [x] make sure we are compatible in the sense of https://github.com/kubernetes/kubernetes/blob/master/docs/devel/api_changes.md
- [x] test e2e with rkt: it only works with kubenet, not with no-op network plugin. The later has no sysctl support.
- ~~[ ] add RunC implementation~~ (~~if that is already in kube,~~ it isn't)
- [x] update whitelist
- [x] switch PSC fields to annotations
- [x] switch PSP fields to annotations
- [x] decide about `--experimental-whitelist-sysctl` flag to be additive or absolute
- [x] decide whether to add a sysctl node whitelist annotation
### Release notes:
```release-note
The pod annotation `security.alpha.kubernetes.io/sysctls` now allows customization of namespaced and well isolated kernel parameters (sysctls), starting with `kernel.shm_rmid_forced`, `net.ipv4.ip_local_port_range`, `net.ipv4.tcp_max_syn_backlog` and `net.ipv4.tcp_syncookies` for Kubernetes 1.4.
The pod annotation `security.alpha.kubernetes.io/unsafeSysctls` allows customization of namespaced sysctls where isolation is unclear. Unsafe sysctls must be enabled at-your-own-risk on the kubelet with the `--experimental-allowed-unsafe-sysctls` flag. Future versions will improve on resource isolation and more sysctls will be considered safe.
```
Automatic merge from submit-queue
Add admission controller for default storage class.
The admission controller adds a default class to PVCs that do not require any
specific class. This way, users (=PVC authors) do not need to care about
storage classes, administrator can configure a default one and all these PVCs
that do not care about class will get the default one.
The marker of default class is annotation "volume.beta.kubernetes.io/storage-class", which must be set to "true" to work. All other values (or missing annotation) makes the class non-default.
Based on @thockin's code, added tests and made it not to reject a PVC when no class is marked as default.
.
@kubernetes/sig-storage
Automatic merge from submit-queue
AppArmor PodSecurityPolicy support
Implements the AppArmor PodSecurityPolicy support based on the alpha API proposed [here](https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/apparmor.md#pod-security-policy)
This implementation deviates from the original proposal in one way: it adds a separate option for specifying a default profile:
```
apparmor.security.alpha.kubernetes.io/defaultProfileName
```
This has several advantages over the original proposal:
- The default is explicit, rather than implicit on the ordering
- The default can be specified without constraining the allowed profiles
- The allowed profiles can be restricted without specifying a default (requires every pod to explicitly set a profile)
The E2E cluster does not currently enable the PodSecurityPolicy, so I will submit E2E tests in a separate PR.
/cc @dchen1107 @pweil- @sttts @jfrazelle @Amey-D
Automatic merge from submit-queue
x509 authenticator: get groups from subject's organization field
Note that the current X509 tests provide a bunch of certs but no private keys or commands to reproduce the testdata, so the new test case isn't added to the certificate chain.
Closes #30260
cc @treed @gtank @mikedanese @deads2k @kubernetes/sig-auth
Automatic merge from submit-queue
only compute delta on non-creating updates
If you're issuing an update that can cause a create, the quota admission charge should be based on the create cost, otherwise you always end up with zero.
@derekwaynecarr ptal, blocker bug.
Automatic merge from submit-queue
ImagePolicyWebhook Admission Controller
<!-- Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, read our contributor guidelines https://github.com/kubernetes/kubernetes/blob/master/CONTRIBUTING.md and developer guide https://github.com/kubernetes/kubernetes/blob/master/docs/devel/development.md
2. If you want *faster* PR reviews, read how: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/faster_reviews.md
3. Follow the instructions for writing a release note: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/pull-requests.md#release-notes
-->
**What this PR does / why we need it**: This is an implementation of the [image provenance proposal](https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/image-provenance.md). It also includes the API definitions by @Q-Lee from https://github.com/kubernetes/kubernetes/pull/30241
**Special notes for your reviewer**:
Please note that this is the first admission controller to make use of the admission controller config file (`--admission-controller-config-file`). I have defined a format for it but we may want to double check it's adequate for future use cases as well.
The format defined is:
```
{
"imagePolicy": {
"kubeConfigFile": "path/to/kubeconfig/for/backend",
"allowTTL": 50, # time in s to cache approval
"denyTTL": 50, # time in s to cache denial
"retryBackoff": 500, # time in ms to wait between retries
"defaultAllow": true # determines behavior if the webhook backend fails
}
}
```
(or yaml)
**Release note**:
<!-- Steps to write your release note:
1. Use the release-note-* labels to set the release note state (if you have access)
2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write `NONE`.
-->
```release-note
Adding ImagePolicyWebhook admission controller.
```
Automatic merge from submit-queue
Quota usage checking ignores unrelated resources
Scenario:
1. Create 4 services
2. Add a quota that limits services to 3
3. Create a pod
Expected result:
pod creation succeeds
Actual result:
pod creation fails stating that services exceed quota.
Fix:
less than or equal check should only verify resources pertinent to request.
Related:
https://bugzilla.redhat.com/show_bug.cgi?id=1367733
Automatic merge from submit-queue
oidc auth provider: don't trim issuer URL
This mirrors a similar side fix for the API server authenticator.
Don't trim the issuer URL provided by the user since OpenID Connect
mandates that this URL exactly matches the URL returned by the
issuer during discovery.
This change only impacts clients attempting to connect to providers that
are non-spec compliant.
No test updates since this is already tested by the go-oidc client
package.
See: https://openid.net/specs/openid-connect-discovery-1_0.html#ProviderConfigurationValidation
Server side fix#29860
Updates #29749
cc @kubernetes/sig-auth @hanikesn
Automatic merge from submit-queue
update comment info for scheduler binding fails
Since the process logic for scheduler binding failed has changed, I think we should update the comment information to avoid make people confused :)
The related issue is #30611.
@wojtek-t What do you think about it ?
Thanks!
Automatic merge from submit-queue
ClusterAutoscaler-friendly scheduler priority function that promotes well-used nodes
It will help cluster autoscaler to put pods on nodes that are unlikely to be deleted soon due to low usage. Otherwise a pod may be frequently kicked from one node to another. A flag that enables it when CA is on will be added in a separate PR.
Fixes: #28842
This mirrors a similar side fix for the API server authenticator.
Don't trim the issuer URL provided by the user since OpenID Connect
mandates that this URL exactly matches the URL returned by the
issuer during discovery.
Not test updates since this is already tested by the go-oidc client
package.
See: https://openid.net/specs/openid-connect-discovery-1_0.html#ProviderConfigurationValidation
Convert single GV and lists of GVs into an interface that can handle
more complex scenarios (everything internal, nothing supported). Pass
the interface down into conversion.
The admission controller adds a default class to PVCs that do not require any
specific class. This way, users (=PVC authors) do not need to care about
storage classes, administrator can configure a default one and all these PVCs
that do not care about class will get the default one.
Automatic merge from submit-queue
Add annotations to the PodSecurityPolicy Provider interface
@pweil- is this what you were thinking in terms of API changes? I really like to avoid functions with more than 2 return values, but couldn't think of a cleaner approach in this case.
Automatic merge from submit-queue
Fix code generators-- make scheme building composable
I needed to make some changes to make my other refactoring possible and this got rather large.
We now provide a "SchemeBuilder" to help all of the api packages provide their scheme-building functions (addKnownTypes and friends) in a standardized way. This also allows generated deepcopies & conversions to be entirely self contained, the project will now build without them being present (as they can add themselves to the SchemeBuilder). (Although if you actually build without them, you will get reduced performance!)
Previously, there was no way to construct your own runtime.Scheme (e.g., to test), you had to use the api.Scheme object, which has all sorts of non-hermetic cruft in it. Now you can get everything from a package by calling the scheme builder's AddToScheme, including the generated functions, if they are present.
Next steps are to allow for declaring dependencies, and to standardize the registration & install code. (#25434)
<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.kubernetes.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.kubernetes.io/reviews/kubernetes/kubernetes/25526)
<!-- Reviewable:end -->
Automatic merge from submit-queue
Cut the client repo, staging it in the main repo
Tracking issue: #28559
ref: https://github.com/kubernetes/kubernetes/pull/25978#issuecomment-232710174
This PR implements the plan a few of us came up with last week for cutting client into its own repo:
1. creating "_staging" (name is tentative) directory in the main repo, using a script to copy the client and its dependencies to this directory
2. periodically publishing the contents of this staging client to k8s.io/client-go repo
3. converting k8s components in the main repo to use the staged client. They should import the staged client as if the client were vendored. (i.e., the import line should be `import "k8s.io/client-go/<pacakge name>`). This requirement is to ease step 4.
4. In the future, removing the staging area, and vendoring the real client-go repo.
The advantage of having the staging area is that we can continuously run integration/e2e tests with the latest client repo and the latest main repo, without waiting for the client repo to be vendored back into the main repo. This staging area will exist until our test matrix is vendoring both the client and the server.
In the above plan, the tricky part is step 3. This PR achieves it by creating a symlink under ./vendor, pointing to the staging area, so packages in the main repo can refer to the client repo as if it's vendored. To prevent the godep tool from messing up the staging area, we export the staged client to GOPATH in hack/godep-save.sh so godep will think the client packages are local and won't attempt to manage ./vendor/k8s.io/client-go.
This is a POC. We'll rearrange the directory layout of the client before merge.
@thockin @lavalamp @bgrant0607 @kubernetes/sig-api-machinery
<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.kubernetes.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.kubernetes.io/reviews/kubernetes/kubernetes/29147)
<!-- Reviewable:end -->
Automatic merge from submit-queue
remove useless value copy
Copy something to values in parameters won't change them in go. So, remove it to avoid making people confused.
Automatic merge from submit-queue
SchedulerExtender: add failedPredicateMap in Filter() returns
Fix#25797. modify extender.Filter for adding extenders information to “failedPredicateMap” in findNodesThatFit.
When all the filtered nodes that passed "predicateFuncs" don’t pass the extenders filter, the failedPredicateMap hasn’t the extenders information, should add it, I think. So when the length of the “filteredNodes.Items” is 0, we can know the integral information. (The length of the “filteredNodes.Items” is 0, may be because the extenders filter failed.)
Automatic merge from submit-queue
Run goimport for the whole repo
While removing GOMAXPROC and running goimports, I noticed quite a lot of other files also needed a goimport format. Didn't commit `*.generated.go`, `*.deepcopy.go` or files in `vendor`
This is more for testing if it builds.
The only strange thing here is the gopkg.in/gcfg.v1 => github.com/scalingdata/gcfg replace.
cc @jfrazelle @thockin
Automatic merge from submit-queue
Move NamespaceLifecycle to use shared informers
This was a follow-up to https://github.com/kubernetes/kubernetes/pull/29634
Moves the `NamespaceLifecycle` plug-in to a shared infomer cache.
/cc @kubernetes/rh-cluster-infra @deads2k @hodovska
Automatic merge from submit-queue
oidc authentication plugin: don't trim issuer URLs with trailing slashes
The issuer URL passed to the plugin must identically match the issuer
URL returned by OpenID Connect discovery. However, the plugin currently
trims all trailing slashes from issuer URLs, causing a mismatch. Since
the go-oidc client already handles this case correctly, don't trim the
path.
Closes#29749
cc @hanikesn @kubernetes/sig-auth
Automatic merge from submit-queue
Prepare for using "ControllerRef" in scheduler
This is part of a PR that I already have to avoid a bunch of rebases in the future (controller ref probably won't happen in 1.4 release).
@davidopp
The issuer URL passed to the plugin must identically match the issuer
URL returned by OpenID Connect discovery. However, the plugin currently
trims all trailing slashes from issuer URLs, causing a mismatch. Since
the go-oidc client already handles this case correctly, don't trim the
path.
Automatic merge from submit-queue
Initial support for pod eviction based on disk
This PR adds the following:
1. node reports disk pressure condition based on configured thresholds
1. scheduler does not place pods on nodes reporting disk pressure
1. kubelet will not admit any pod when it reports disk pressure
1. kubelet ranks pods for eviction when low on disk
1. kubelet evicts greediest pod
Follow-on PRs will need to handle:
1. integrate with new image gc PR (https://github.com/kubernetes/kubernetes/pull/27199)
1. container gc policy should always run (will not be launched from eviction, tbd who does that)
1. this means kill pod is fine for all eviction code paths since container gc will remove dead container
1. min reclaim support will just poll summary provider (derek will do follow-on)
1. need to know if imagefs is same device as rootfs from summary (derek follow-on)
/cc @vishh @kubernetes/sig-node
Automatic merge from submit-queue
Quota was not counting services with multiple nodeports properly
```release-note
If a service of type node port declares multiple ports, quota on "services.nodeports" will charge for each port in the service.
```
Fixes https://github.com/kubernetes/kubernetes/issues/29456
/cc @kubernetes/rh-cluster-infra @sdminonne
Automatic merge from submit-queue
LimitRanger and PodSecurityPolicy need to check more on init containers
Container limits not applied to init containers. HostPorts not checked on podsecuritypolicy
@pweil- @derekwaynecarr
Automatic merge from submit-queue
Allow shareable resources for admission control plugins.
Changes allow admission control plugins to share resources. This is done via new PluginInitialization structure. The structure can be extended for other resources, for now it is an shared informer for namespace plugins (NamespiceLifecycle, NamespaceAutoProvisioning, NamespaceExists).
If a plugins needs some kind of shared resource e.g. client, the client shall be added to PluginInitializer and Wants methods implemented to every plugin which will use it.
Automatic merge from submit-queue
Improve scheduler throughput
Ref #28590
This improves scheduler throughput by another 10-15%.
@davidopp @kubernetes/sig-scheduling
Automatic merge from submit-queue
Remove GOMAXPROCS() calls because they are unnecessary
Now we're setting GOMAXPROCS when every binary starts up, but we don't have to do that anymore, since we've upgraded to Go 1.6
Documentation for it:
> func GOMAXPROCS(n int) int
> GOMAXPROCS sets the maximum number of CPUs that can be executing simultaneously and returns the previous setting. If n < 1, it does not change the current setting. The number of logical CPUs on the local machine can be queried with NumCPU. This call will go away when the scheduler improves.
A simple program to prove it's unnecessary:
```go
package main
import (
"fmt"
"runtime"
)
func main(){
numCPUBefore := runtime.GOMAXPROCS(runtime.NumCPU())
numCPUAfter := runtime.GOMAXPROCS(runtime.NumCPU())
fmt.Println(numCPUBefore, numCPUAfter)
}
```
Output with Go 1.4.2: `1 4`
Output with Go 1.6.2: `4 4`
So I think we should remove calls to GOMAXPROCS now, and it should be pretty straightforward
@thockin @wojtek-t @gmarek @lavalamp @vishh
Automatic merge from submit-queue
Return (bool, error) in Authorizer.Authorize()
Before this change, Authorize() method was just returning an error, regardless of whether the user is unauthorized or whether there is some other unrelated error. Returning boolean with information about user authorization and error (which should be unrelated to the authorization) separately will make it easier to debug.
Fixes#27974
Automatic merge from submit-queue
Func note not consistent with real fun name
File "plugin\pkg\webhook.go", line #41 :
"// New creates a new GenericWebhook from the provided kubeconfig file."
Here "New" not consistant with real fun name "NewGenericWebhook" in line #42 :
"func NewGenericWebhook(kubeConfigFile string, groupVersions []unversioned.GroupVersion, initialBackoff time.Duration) (*GenericWebhook, error) {"
Before this change, Authorize() method was just returning an error,
regardless of whether the user is unauthorized or whether there
is some other unrelated error. Returning boolean with information
about user authorization and error (which should be unrelated to
the authorization) separately will make it easier to debug.
Fixes#27974
Automatic merge from submit-queue
authorize based on user.Info
Update the `authorization.Attributes` to use the `user.Info` instead of discrete getters for each piece.
@kubernetes/sig-auth
Automatic merge from submit-queue
First step of optimizing PodAffinity priority function
Ref #26144
This is obviously only a first step - I will continue working on this code. However, this is changing the general scheme of computations to what is described in: https://github.com/kubernetes/kubernetes/issues/26144#issuecomment-232612384
Automatic merge from submit-queue
Optimise the process of the CalculateSpreadPriority in selector_spreading.go
It had better inspect if the nodeLister is normal first in the CalculateSpreadPriority in selector_spreading.go. If the nodeLister.List return error, the function return directly, not need deal the serviceLister and controllerLister and replicaSetLister.
Automatic merge from submit-queue
rbac authorizer: cleanups to rule evaluation for non-resource URLs
An few oversights in the RBAC authorizer. Fixes#28291 and permits non-resource URLs to use stars in the path. E.g. ("/apis/*").
cc @liggitt @kubernetes/sig-auth
Automatic merge from submit-queue
Add hooks for cluster health detection
Separate a function that decides if zone is healthy. First real commit for preventing massive pod eviction.
Ref. #28832
cc @davidopp
Automatic merge from submit-queue
Change storeToNodeConditionLister to return []*api.Node instead of api.NodeList for performance
Currently copies that are made while copying/creating api.NodeList are significant part of scheduler profile, and a bunch of them are made in places, that are not-parallelizable.
Ref #28590
Automatic merge from submit-queue
Add test case to TestPodFitsResources() of scheduler algorithm
File "plugin\pkg\scheduler\algorithm\predicates", function "TestPodFitsResources()", line 199, only provide test case "one resource cpu fits but memory not", it should add test case "one resource memory fits but cpu not".
Automatic merge from submit-queue
Optimize priorities in scheduler
Ref #28590
It's probably easier to review it commit by commit, since those changes are kind of independent from each other.
@davidopp - FYI
Automatic merge from submit-queue
Add meta field to predicate signature to avoid computing the same things multiple times
This PR only uses it to avoid computing QOS of a pod for every node from scratch.
Ref #28590
Automatic merge from submit-queue
allow lock acquisition injection for quota admission
Allows for custom lock acquisition when composing the quota admission controller.
@derekwaynecarr I'm still experimenting to make sure this satisfies the need downstream, but looking for agreement in principle
Automatic merge from submit-queue
getting emailAddress from TLS cert
Kubernetes if using TLS cert to perform authentication will use the CommonName field of the cert as the authenticating user. In https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/auth/authenticator/request/x509/x509.go#L106, alternative methods are defined to use emailAddress or DNSName as the authenticating user. The method that uses the emailAddress is not comprehensive as this information can be encoded in different places of the certificate. This PR fixes this.
Automatic merge from submit-queue
oidc auth plugin: don't hard fail if provider is unavailable
When using OpenID Connect authentication, don't cause the API
server to fail if the provider is unavailable. This allows
installations to run OpenID Connect providers after starting the
API server, a common case when the provider is running on the
cluster itself.
Errors are now deferred to the authenticate method.
cc @sym3tri @erictune @aaronlevy @kubernetes/sig-auth
When using OpenID Connect authentication, don't cause the API
server to fail if the provider is unavailable. This allows
installations to run OpenID Connect providers after starting the
API server, a common case when the provider is running on the
cluster itself.
Errors are now deferred to the authenticate method.
Automatic merge from submit-queue
Add error log for Run function in server.go
When clientcmd.BuildConfigFromFlags and os.Hostname return error, there are no log information in Run function, neither did the upper function(main of scheduler), add it, I suggest.
Automatic merge from submit-queue
scheduler: change phantom pod test from integration into unit test
This is an effort for #24440.
Why this PR?
- Integration test is hard to debug. We could model the test as a unit test similar to [TestSchedulerForgetAssumedPodAfterDelete()](132ebb091a/plugin/pkg/scheduler/scheduler_test.go (L173)). Currently the test is testing expiring case, we can change that to delete.
- Add a test similar to TestSchedulerForgetAssumedPodAfterDelete() to test phantom pod.
- refactor scheduler tests to share the code between TestSchedulerNoPhantomPodAfterExpire() and TestSchedulerNoPhantomPodAfterDelete()
- Decouple scheduler tests from scheduler events: not to use events
Automatic merge from submit-queue
Track object modifications in fake clientset
Fake clientset is used by unit tests extensively but it has some
shortcomings:
- no filtering on namespace and name: tests that want to test objects in
multiple namespaces end up getting all objects from this clientset,
as it doesn't perform any filtering based on name and namespace;
- updates and deletes don't modify the clientset state, so some tests
can get unexpected results if they modify/delete objects using the
clientset;
- it's possible to insert multiple objects with the same
kind/name/namespace, this leads to confusing behavior, as retrieval is
based on the insertion order, but anchors on the last added object as
long as no more objects are added.
This change changes core.ObjectRetriever implementation to track object
adds, updates and deletes.
Some unit tests were depending on the previous (and somewhat incorrect)
behavior. These are fixed in the following few commits.
Automatic merge from submit-queue
[Refactor] Make QoS naming consistent across the codebase
@derekwaynecarr @vishh PTAL. Can one of you please attach a LGTM.
Fake clientset no longer needs to be prepopulated with records: keeping
them in leads to the name conflict on creates. Also, since fake
clientset now respects namespaces, we need to correctly populate them.
Automatic merge from submit-queue
Omit invalid affinity error in admission
Fixes#27645 cc @smarterclayton
Not sure if this is too aggressive, but user should expect failure if they disable validation after all.
Automatic merge from submit-queue
[Refactor] QOS to have QOS Class type for QoS classes
This PR adds a QOSClass type and initializes QOSclass constants for the three QoS classes.
It would be good to use this in all future QOS related features.
This would be good to have for the (Pod level cgroups isolation proposal)[https://github.com/kubernetes/kubernetes/pull/26751] that i am working on aswell.
@vishh PTAL
Signed-off-by: Buddha Prakash <buddhap@google.com>
Automatic merge from submit-queue
Fix #25606: Add the length detection of the "predicateFuncs" in generic_scheduler.go
Fix #25606
The PR add the length detection of the "predicateFuncs" for "findNodesThatFit" function of generic_scheduler.go.
In “findNodesThatFit” function, if the length of the "predicateFuncs" parameter is 0, it can set filtered equals nodes.Items, and needn't to traverse the nodes.Items.
Automatic merge from submit-queue
plugin/pkg/auth/authorizer/webhook: log request errors
Currently the API server only checks the errors returned by an
authorizer plugin, it doesn't return or log them[0]. This makes
incorrectly configuring the wehbook authorizer plugin extremely
difficult to debug.
Add a logging statement if the request to the remove service fails
as this indicates misconfiguration.
[0] https://goo.gl/9zZFv4
<!-- Reviewable:start -->
---
This change is [<img src="http://reviewable.k8s.io/review_button.svg" height="35" align="absmiddle" alt="Reviewable"/>](http://reviewable.k8s.io/reviews/kubernetes/kubernetes/24678)
<!-- Reviewable:end -->
Automatic merge from submit-queue
golint fixes for AWS cloudprovider
Among other things, golint doesn't like receivers that are inconsistently named or called "self". Or structs named aws.AWSservices, aws.AWSCloud, etc.
Automatic merge from submit-queue
Rephrase 'pv not found in cache' warnings.
When kubelet starts a pod that refers to non-existing PV, PVC or Node, it should clearly show that the requested element does not exist.
Previous `PersistentVolumeClaim 'default/ceph-claim-wm' is not in cache` looks like random kubelet hiccup, while `PersistentVolumeClaim 'default/ceph-claim-wm' not found` suggests that the object may not exist at all and it might be an user error.
Fixes#27523
Automatic merge from submit-queue
AWS/GCE: Spread PetSet volume creation across zones, create GCE volumes in non-master zones
Long term we plan on integrating this into the scheduler, but in the
short term we use the volume name to place it onto a zone.
We hash the volume name so we don't bias to the first few zones.
If the volume name "looks like" a PetSet volume name (ending with
-<number>) then we use the number as an offset. In that case we hash
the base name.
When kubelet starts a pod that refers to non-existing PV, PVC or Node, it
should clearly show that the requested element does not exist.
Previous "PersistentVolumeClaim 'default/ceph-claim-wm' is not in cache"
looks like random kubelet hiccup, while "PersistentVolumeClaim
'default/ceph-claim-wm' not found" suggests that the object may not exist at
all and it might be an user error.
Fixes#27523
Automatic merge from submit-queue
add unit and integration tests for rbac authorizer
This PR adds lots of tests for the RBAC authorizer.
The plan over the next couple days is to add a lot more test cases.
Updates #23396
cc @erictune
We had a long-lasting bug which prevented creation of volumes in
non-master zones, because the cloudprovider in the volume label
admission controller is not initialized with the multizone setting
(issue #27656).
This implements a simple workaround: if the volume is created with the
failure-domain zone label, we look for the volume in that zone. This is
more efficient, avoids introducing a new semantic, and allows users (and
the dynamic provisioner) to create volumes in non-master zones.
Fixes#27657
Automatic merge from submit-queue
Considering all nodes for the scheduler cache to allow lookups
Fixes the actual issue that led me to create https://github.com/kubernetes/kubernetes/issues/22554
Currently the nodes in the cache provided to the predicates excludes the unschedulable nodes using field level filtering for the watch results. This results in the above issue as the `ServiceAffinity` predicate uses the cached node list to look up the node metadata for a peer pod (another pod belonging to the same service). Since this peer pod could be currently hosted on a node that is currently unschedulable, the lookup could potentially fail, resulting in the pod failing to be scheduled.
As part of the fix, we are now including all nodes in the watch results and excluding the unschedulable nodes using `NodeCondition`
@derekwaynecarr PTAL
Currently the API server only checks the errors returned by an
authorizer plugin, it doesn't return or log them[0]. This makes
incorrectly configuring the wehbook authorizer plugin extremely
difficult to debug.
Add a logging statement if the request to the remove service fails
as this indicates misconfiguration.
[0] https://goo.gl/9zZFv4
Automatic merge from submit-queue
scheduler: remove unused random generator
The way scheduler selecting host has been changed to round-robin.
Clean up leftover.
Automatic merge from submit-queue
Add a NodeCondition "NetworkUnavaiable" to prevent scheduling onto a node until the routes have been created
This is new version of #26267 (based on top of that one).
The new workflow is:
- we have an "NetworkNotReady" condition
- Kubelet when it creates a node, it sets it to "true"
- RouteController will set it to "false" when the route is created
- Scheduler is scheduling only on nodes that doesn't have "NetworkNotReady ==true" condition
@gmarek @bgrant0607 @zmerlynn @cjcullen @derekwaynecarr @danwinship @dcbw @lavalamp @vishh
Automatic merge from submit-queue
reduce conflict retries
Eliminates quota admission conflicts due to latent caches on the same API server.
@derekwaynecarr
Automatic merge from submit-queue
plumb Update resthandler to allow old/new comparisons in admission
Rework how updated objects are passed to rest storage Update methods (first pass at https://github.com/kubernetes/kubernetes/pull/23928#discussion_r61444342)
* allows centralizing precondition checks (uid and resourceVersion)
* allows admission to have the old and new objects on patch/update operations (sets us up for field level authorization, differential quota updates, etc)
* allows patch operations to avoid double-GETting the object to apply the patch
Overview of important changes:
* pkg/api/rest/rest.go
* changes `rest.Update` interface to give rest storage an `UpdatedObjectInfo` interface instead of the object directly. To get the updated object, the storage must call `UpdatedObject()`, passing in the current object
* pkg/api/rest/update.go
* provides a default `UpdatedObjectInfo` impl
* passes a copy of the updated object through any provided transforming functions and returns it when asked
* builds UID preconditions from the updated object if they can be extracted
* pkg/apiserver/resthandler.go
* Reworks update and patch operations to give old objects to admission
* pkg/registry/generic/registry/store.go
* Calls `UpdatedObject()` inside `GuaranteedUpdate` so it can provide the old object
Todo:
- [x] Update rest.Update interface:
* Given the name of the object being updated
* To get the updated object data, the rest storage must pass the current object (fetched using the name) to an `UpdatedObject(ctx, oldObject) (newObject, error)` func. This is typically done inside a `GuaranteedUpdate` call.
- [x] Add old object to admission attributes interface
- [x] Update resthandler Update to move admission into the UpdatedObject() call
- [x] Update resthandler Patch to move the patch application and admission into the UpdatedObject() call
- [x] Add resttest tests to make sure oldObj is correctly passed to UpdatedObject(), and errors propagate back up
Follow-up:
* populate oldObject in admission for delete operations?
* update quota plugin to use `GetOldObject()` in admission attributes
* admission plugin to gate ownerReference modification on delete permission
* Decide how to handle preconditions (does that belong in the storage layer or in the resthander layer?)
Automatic merge from submit-queue
Introduce node memory pressure condition to scheduler
Following the work done by @derekwaynecarr at https://github.com/kubernetes/kubernetes/pull/21274, introducing memory pressure predicate for scheduler.
Missing:
* write down unit-test
* test the implementation
At the moment this is a heads up for further discussion how the new node's memory pressure condition should be handled in the generic scheduler.
**Additional info**
* Based on [1], only best effort pods are subject to filtering.
* Based on [2], best effort pods are those pods "iff requests & limits are not specified for any resource across all containers".
[1] 542668cc79/docs/proposals/kubelet-eviction.md (scheduler)
[2] https://github.com/kubernetes/kubernetes/pull/14943
Automatic merge from submit-queue
Use protobufs by default to communicate with apiserver (still store JSONs in etcd)
@lavalamp @kubernetes/sig-api-machinery
Automatic merge from submit-queue
Cache Webhook Authentication responses
Add a simple LRU cache w/ 2 minute TTL to the webhook authenticator.
Kubectl is a little spammy, w/ >= 4 API requests per command. This also prevents a single unauthenticated user from being able to DOS the remote authenticator.
The PR add the length detection of the "predicateFuncs" for "findNodesThatFit" function of generic_scheduler.go.
In “findNodesThatFit” function, if the length of the "predicateFuncs" parameter is 0, it can set filtered equals nodes.Items, and needn't to traverse the nodes.Items.
It should reduce the resource data after finding the pod in the pods, because perhaps no corresponding pod in the pods of the node, at this time it shouldn't reduce the resource data of the node.
* Use an interface for OIDC Client, so that we're testing the behavior
of the client, not the go-oidc package itself
* add backoff and retry when server rejects token
This commit handles:
* Passing ID Token as Bearer token
* Refreshing of tokens using refresh-tokens
* Persisting refreshed tokens
* ability to add arbitrary extra scopes via config
* this is what enables the cross-client/azp stuff
Automatic merge from submit-queue
Make IsValidLabelValue return error strings
Part of the larger validation PR, broken out for easier review and merge. Builds on previous PRs in the series.
Automatic merge from submit-queue
Improve fatal error description in plugins.go of scheduler
The PR add more information for the fatal error in plugins.go of scheduler.
Automatic merge from submit-queue
Make IsQualifiedName return error strings
Part of the larger validation PR, broken out for easier review and merge.
@lavalamp FYI, but I know you're swamped, too.
In RegisterCustomFitPredicate, when policy.Argument is nil and fitPredicateMap has the policy.Name, it can return the policy.Name directly. Subsequent operations are redundant.
Automatic merge from submit-queue
WIP v0 NVIDIA GPU support
```release-note
* Alpha support for scheduling pods on machines with NVIDIA GPUs whose kubelets use the `--experimental-nvidia-gpus` flag, using the alpha.kubernetes.io/nvidia-gpu resource
```
Implements part of #24071 for #23587
I am not familiar with the scheduler enough to know what to do with the scores. Mostly punting for now.
Missing items from the implementation plan: limitranger, rkt support, kubectl
support and docs
cc @erictune @davidopp @dchen1107 @vishh @Hui-Zhi @gopinatht
Automatic merge from submit-queue
Automatically create the kube-system namespace
At the same time we ensure that the `default` namespace is present, it also creates `kube-system` if it doesn't exist.
`kube-system` will now exist from the beginning, and will be recreated every 10s if deleted, in the same manner as the `default` ns
This makes UX much better, no need for `kubectl`ing a `kube-system.yaml` file anymore for a function that is essential to Kubernetes (addons). For instance, this makes dashboard deployment much easier when there's no need to check for the `kube-system` ns first.
A follow up in the future may remove places where logic to manually create the kube-system namespace is present.
Also fixed a small bug where `CreateNamespaceIfNeeded` ignored the `ns` parameter and was hardcoded to `api.NamespaceDefault`.
@davidopp @lavalamp @thockin @mikedanese @bryk @cheld @fgrzadkowski @smarterclayton @wojtek-t @dlorenc @vishh @dchen1107 @bgrant0607 @roberthbailey
<!-- Reviewable:start -->
---
This change is [<img src="http://reviewable.k8s.io/review_button.svg" height="35" align="absmiddle" alt="Reviewable"/>](http://reviewable.k8s.io/reviews/kubernetes/kubernetes/25196)
<!-- Reviewable:end -->
Automatic merge from submit-queue
Add pod condition PodScheduled to detect situation when scheduler tried to schedule a Pod, but failed
Set `PodSchedule` condition to `ConditionFalse` in `scheduleOne()` if scheduling failed and to `ConditionTrue` in `/bind` subresource.
Ref #24404
@mml (as it seems to be related to "why pending" effort)
<!-- Reviewable:start -->
---
This change is [<img src="http://reviewable.k8s.io/review_button.svg" height="35" align="absmiddle" alt="Reviewable"/>](http://reviewable.k8s.io/reviews/kubernetes/kubernetes/24459)
<!-- Reviewable:end -->
Automatic merge from submit-queue
Webhook Token Authenticator
Add a webhook token authenticator plugin to allow a remote service to make authentication decisions.
Automatic merge from submit-queue
Sort resources in quota errors to avoid duplicate events
Bugzilla:
https://bugzilla.redhat.com/show_bug.cgi?id=1333122
Errors describing why a request was rejected to quota would get variable responses (cpu=x,memory=y or memory=x,cpu=y) which caused duplicate events for the same root cause.
/cc @ncdc @jwforres
Implements part of #24071
I am not familiar with the scheduler enough to know what to do with the scores. Punting for now.
Missing items from the implementation plan: limitranger, rkt support, kubectl
support and user docs
Automatic merge from submit-queue
Support persisting config from kubecfg AuthProvider plugins
Plumbs through an interface to the plugin that can persist a `map[string]string` config for just that plugin. Also adds `config` to the AuthProvider serialization type, and `Login()` to the AuthProvider plugin interface.
Modified the gcp AuthProvider to cache short-term access tokens in the kubecfg file.
Builds on #23066
@bobbyrullo @deads2k @jlowdermilk @erictune
Automatic merge from submit-queue
add namespace index for cache
@wojtek-t
Implement in this approach make the change of lister.go small, but we should replace all `NewInformer()` to `NewIndexInformer()`, even when someone not want to filter by namespace(eg. gc_controller and scheduler). Any suggestion?
The codec factory should support two distinct interfaces - negotiating
for a serializer with a client, vs reading or writing data to a storage
form (etcd, disk, etc). Make the EncodeForVersion and DecodeToVersion
methods only take Encoder and Decoder, and slight refactoring elsewhere.
In the storage factory, use a content type to control what serializer to
pick, and use the universal deserializer. This ensures that storage can
read JSON (which might be from older objects) while only writing
protobuf. Add exceptions for those resources that may not be able to
write to protobuf (specifically third party resources, but potentially
others in the future).
Automatic merge from submit-queue
fully qualify admission resources and kinds
Fully qualifies the `Kind` and `Resource` fields for admission attributes. The information was getting filtered at the `RESTHandler` before.
@derekwaynecarr
Automatic merge from submit-queue
Store node information in NodeInfo
This is significantly improving scheduler throughput.
On 1000-node cluster:
- empty cluster: ~70pods/s
- full cluster: ~45pods/s
Drop in throughput is mostly related to priority functions, which I will be looking into next (I already have some PR #24095, but we need for more things before).
This is roughly ~40% increase.
However, we still need better understanding of predicate function, because in my opinion it should be even faster as it is now. I'm going to look into it next week.
@gmarek @hongchaodeng @xiang90
Automatic merge from submit-queue
Add Subresource & Name to webhook authorizer.
Pass through the Subresource and Name fields from the `authorization.Attributes` to the `SubjectAccessReviewSpec.ResourceAttributes`.
Automatic merge from submit-queue
schedulercache: remove bind() from AssumePod
Due to #24197, we make bind() asynchronous and don't need it anymore.
Submit this PR to clean it up.
Automatic merge from submit-queue
Client auth provider plugin framework
Allows client plugins to modify the underlying transport to, for example, add custom authorization headers.
Automatic merge from submit-queue
Move predicates into library
This PR tries to implement #12744
Any suggestions/ideas are welcome. @davidopp
current state: integration test fails if including podCount check in Kubelet.
DONE:
1. refactor all predicates: predicates return fitOrNot(bool) and error(Error) in which the latter is of type PredicateFailureError or InsufficientResourceError
2. GeneralPredicates() is a predicate function, which includes serveral other predicate functions (PodFitsResource, PodFitsHost, PodFitsHostPort). It is registered as one of the predicates in DefaultAlgorithmProvider, and is also called in canAdmitPod() in Kubelet and should be called by other components (like rescheduler, etc if necessary. See discussion in issue #12744
TODO:
1. determine which predicates should be included in GeneralPredicates()
2. separate GeneralPredicates() into: a.) GeneralPredicatesEvictPod() and b.) GeneralPredicatesNotEvictPod()
3. DaemonSet should use GeneralPredicates()
DONE:
1. refactor all predicates: predicates return fitOrNot(bool) and error(Error) in which the latter is of type
PredicateFailureError or InsufficientResourceError. (For violation of either MaxEBSVolumeCount or
MaxGCEPDVolumeCount, returns one same error type as ErrMaxVolumeCountExceeded)
2. GeneralPredicates() is a predicate function, which includes serveral other predicate functions (PodFitsResource,
PodFitsHost, PodFitsHostPort). It is registered as one of the predicates in DefaultAlgorithmProvider, and
is also called in canAdmitPod() in Kubelet and should be called by other components (like rescheduler, etc)
if necessary. See discussion in issue #12744
3. remove podNumber check from GeneralPredicates
4. HostName is now verified in Kubelet's canAdminPod(). add TestHostNameConflicts in kubelet_test.go
5. add getNodeAnyWay() method in Kubelet to get node information in standaloneMode
TODO:
1. determine which predicates should be included in GeneralPredicates()
2. separate GeneralPredicates() into:
a. GeneralPredicatesEvictPod() and
b. GeneralPredicatesNotEvictPod()
3. DaemonSet should use GeneralPredicates()
AWS has soft support limit for 40 attached EBS devices. Assuming there is just
one root device, use the rest for persistent volumes.
The devices will have name /dev/xvdba - /dev/xvdcm, leaving /dev/sda - /dev/sdz
to the system.
Also, add better error handling and propagate error
"Too many EBS volumes attached to node XYZ" to a pod.
Custom quota evaluators may need to query a target namespace of an input
object during quota admission check. For this, namespace needs to be
known.
Signed-off-by: Michal Minar <miminar@redhat.com>
The provider config has changed a little bit in go-oidc. It is more
complete and now throws errors when unmarshaling provider configs
that are missing required fields (as defined by the OpenID Connect
Discovery spec).
Update the oidc plugin to use the new type.
This is a first-aid bandage to let admission controller ignore persistent
volumes that are being provisioned right now and thus may not exist in
external cloud infrastructure yet.
Instead of sorting hosts by score, find max score and choose one of
the hosts with max score directly. Saves a little time on sorting and
avoids extra copying of HostPriorityList entries.
Test had to be updated as one of the test cases relied on having a
stable order from pre-sorting.
I can't revert with github which says "Sorry, this pull request couldn’t be
reverted automatically. It may have already been reverted, or the content may
have changed since it was merged."
Reverts commit: 0c191e787b
We are (sadly) using a copy-and-paste of the GCE PD code for AWS EBS.
This code hasn't been updated in a while, and it seems that the GCE code
has some code to make volume mounting more robust that we should copy.
For certain volume types (e.g. AWS EBS or GCE PD), a limitted
number of such volumes can be attached to a given node. This commit
introduces a predicate with allows cluster admins to cap
the maximum number of volumes matching a particular type attached to a
given node.
The volume type is configurable by passing a pair of filter functions,
and the maximum number of such volumes is configurable to allow node
admins to reserve a certain number of volumes for system use.
By default, the predicate is exposed as MaxEBSVolumeCount and
MaxGCEPDVolumeCount (for AWS ElasticBlocKStore and GCE PersistentDisk
volumes, respectively), each of which can be configured using the
`KUBE_MAX_PD_VOLS` environment variable.
Fixes#7835
Combine the fields that will be used for content transformation
(content-type, codec, and group version) into a single struct in client,
and then pass that struct into the rest client and request. Set the
content-type when sending requests to the server, and accept the content
type as primary.
Will form the foundation for content-negotiation via the client.
Remove Codec from versionInterfaces in meta (RESTMapper is now agnostic
to codec and serialization). Register api/latest.Codecs as the codec
factory and use latest.Codecs.LegacyCodec(version) as an equvialent to
the previous codec.
For AWS EBS, a volume can only be attached to a node in the same AZ.
The scheduler must therefore detect if a volume is being attached to a
pod, and ensure that the pod is scheduled on a node in the same AZ as
the volume.
So that the scheduler need not query the cloud provider every time, and
to support decoupled operation (e.g. bare metal) we tag the volume with
our placement labels. This is done automatically by means of an
admission controller on AWS when a PersistentVolume is created backed by
an EBS volume.
Support for tagging GCE PVs will follow.
Pods that specify a volume directly (i.e. without using a
PersistentVolumeClaim) will not currently be scheduled correctly (i.e.
they will be scheduled without zone-awareness).
Public utility methods and JWT parsing, and controller specific logic.
Also remove the coupling between ServiceAccountTokenGetter and the
authenticator class.
1. Name default scheduler with name `kube-scheduler`
2. The default scheduler only schedules the pods meeting the following condition:
- the pod has no annotation "scheduler.alpha.kubernetes.io/name: <scheduler-name>"
- the pod has annotation "scheduler.alpha.kubernetes.io/name: kube-scheduler"
update gofmt
update according to @david's review
run hack/test-integration.sh, hack/test-go.sh and local e2e.test
To improve the throughput of current scheduler, we can do
a simple optimization by calcluating priorities in parallel.
This doubles the throughput of density test, which has the default
config with 3 priority funcs (the spreading one does not actually
consume any computation time). It matches the expectation.
The "Combined requested resources" message becomes excessive as
the cluster fills up, drop it down to V(2)
Put an explicit V(2) on the only other scheduler Infof call that didn't
have V specified already.
Set the out of disk node condition to unknown in the node controller if
the kubelet does not report its node condition in a long time. Update
node controller unit tests.
Implement a node condition predicate function that checks if a given
node satisfies the conditions defined by the predicate and if it
does, use that node for scheduling pods. The predicate function takes
both NodeReady and NodeOutOfDisk into consideration to determine if a
node is fit for scheduling pods.
The predicate is then passed to the node lister in the scheduler factory
so that the node lister can run the predicate function on the nodes when
schedling pods thereby omitting nodes that does not satisfy the
predicate.
Also update listers test.
Default to hardcodes for components that had them, and 5.0 qps, 10 burst
for those that relied on client defaults
Unclear if maybe it'd be better to just assume these are set as part of
the incoming kubeconfig. For now just exposing them as flags since it's
easier for me to manually tweak.
.State.Phase is read and written by multiple goroutines as reported by `godep
go test -race` on Go 1.5.1. Adding the mutex around the object fixes the
issue.
Add a HostIPC field to the Pod Spec to create containers sharing
the same ipc of the host.
This feature must be explicitly enabled in apiserver using the
option host-ipc-sources.
Signed-off-by: Federico Simoncelli <fsimonce@redhat.com>
A lot of packages use StringSet, but they don't use anything else from
the util package. Moving StringSet into another package will shrink
their dependency trees significantly.
This commit merges the NamespaceExists admission controller
into the NamespaceLifecycle admission controller.
New tests were added to the NamespaceLifecycle addmission controller
tests, and the test case was fixed so that it runs without panicing.
Additionally, the NamespaceExists admission controller was marked as
deprecated in the docs.
Closes#12053
Also add related new flags to apiserver:
"--oidc-issuer-url", "--oidc-client-id", "--oidc-ca-file", "--oidc-username-claim",
to enable OpenID Connect authentication.
pflag can handle IP addresses so use the pflag code instead of doing it
ourselves. This means our code just uses net.IP and we don't have all of
the useless casting back and forth!
These messages are only useful if you want to debug a particular
scheduler assigment, and they are extremely verbose-- they each print
out a line per host per assignment. Let's try to keep our log messages
linear in the number of assigments.
This commit wires together the graceful delete option for pods
on the Kubelet. When a pod is deleted on the API server, a
grace period is calculated that is based on the
Pod.Spec.TerminationGracePeriodInSeconds, the user's provided grace
period, or a default. The grace period can only shrink once set.
The value provided by the user (or the default) is set onto metadata
as DeletionGracePeriod.
When the Kubelet sees a pod with DeletionTimestamp set, it uses the
value of ObjectMeta.GracePeriodSeconds as the grace period
sent to Docker. When updating status, if the pod has DeletionTimestamp
set and all containers are terminated, the Kubelet will update the
status one last time and then invoke Delete(pod, grace: 0) to
clean up the pod immediately.
Having no nodes in the cluster is unusual and is likely a test
environment, and when a pod is deleted there is no need to log
information about our inability to schedule it.
--master option still supported.
--kubeconfig option added to kube-proxy,
kube-scheduler, and kube-controller-manager
binaries.
Kube-proxy now always makes some kind of API
source, since that is its only kind of config.
Warn if it is using a default client, which probably won't work.
Uses the clientcmd builder.
Standardize how our fakes are used so that a test case can use a
simpler mechanism for providing large, complex data sets, as well
as represent queries over time.
It currently is impossible to use two healthz handlers on different
ports in the same process. This removes the global variables in favor
of requiring the consumer to specify all health checks up front.
Allows POST to create a binding as a child. Also refactors internal
and v1beta3 Binding to be more generic (so that other resources can
support Bindings).
