github/kubernetes
4
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge pull request #40796 from wojtek-t/use_node_ttl_in_secret_manager

Browse Source 
Automatic merge from submit-queue (batch tested with PRs 40796, 40878, 36033, 40838, 41210)

Implement TTL controller and use the ttl annotation attached to node in secret manager

For every secret attached to a pod as volume, Kubelet is trying to refresh it every sync period. Currently Kubelet has a ttl-cache of secrets of its pods and the ttl is set to 1 minute. That means that in large clusters we are targetting (5k nodes, 30pods/node), given that each pod has a secret associated with ServiceAccount from its namespaces, and with large enough number of namespaces (where on each node (almost) every pod is from a different namespace), that resource in ~30 GETs to refresh all secrets every minute from one node, which gives ~2500QPS for GET secrets to apiserver.

Apiserver cannot keep up with it very easily.

Desired solution would be to watch for secret changes, but because of security we don't want a node watching for all secrets, and it is not possible for now to watch only for secrets attached to pods from my node.

So as a temporary solution, we are introducing an annotation that would be a suggestion for kubelet for the TTL of secrets in the cache and a very simple controller that would be setting this annotation based on the cluster size (the large cluster is, the bigger ttl is). 
That workaround mean that only very local changes are needed in Kubelet, we are creating a well separated very simple controller, and once watching "my secrets" will be possible it will be easy to remove it and switch to that. And it will allow us to reach scalability goals.

@dchen1107 @thockin @liggitt
This commit is contained in:
Kubernetes Submit Queue
2017-02-10 00:04:44 -08:00
committed by GitHub
parent e1720af730 dcf8a85fdf
commit 8188c3cca4
16 changed files with 982 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
cmd/kube-controller-manager/app/BUILD
View File

@@ -64,6 +64,7 @@ go_library(
"//pkg/controller/service:go_default_library",
"//pkg/controller/serviceaccount:go_default_library",
"//pkg/controller/statefulset:go_default_library",
"//pkg/controller/ttl:go_default_library",
"//pkg/controller/volume/attachdetach:go_default_library",
"//pkg/controller/volume/persistentvolume:go_default_library",
"//pkg/features:go_default_library",

1
cmd/kube-controller-manager/app/controllermanager.go
View File

@@ -287,6 +287,7 @@ func newControllerInitializers() map[string]InitFunc {
controllers["statefuleset"] = startStatefulSetController
controllers["cronjob"] = startCronJobController
controllers["certificatesigningrequests"] = startCSRController
controllers["ttl"] = startTTLController
return controllers
}

9
cmd/kube-controller-manager/app/core.go
View File

@@ -40,6 +40,7 @@ import (
replicationcontroller "k8s.io/kubernetes/pkg/controller/replication"
resourcequotacontroller "k8s.io/kubernetes/pkg/controller/resourcequota"
serviceaccountcontroller "k8s.io/kubernetes/pkg/controller/serviceaccount"
ttlcontroller "k8s.io/kubernetes/pkg/controller/ttl"
quotainstall "k8s.io/kubernetes/pkg/quota/install"
)
@@ -141,6 +142,14 @@ func startServiceAccountController(ctx ControllerContext) (bool, error) {
return true, nil
}
func startTTLController(ctx ControllerContext) (bool, error) {
go ttlcontroller.NewTTLController(
ctx.NewInformerFactory.Core().V1().Nodes(),
ctx.ClientBuilder.ClientOrDie("ttl-controller"),
).Run(5, ctx.Stop)
return true, nil
}
func startGarbageCollectorController(ctx ControllerContext) (bool, error) {
if !ctx.Options.EnableGarbageCollector {
return false, nil