f5c29f51fa
Automatic merge from submit-queue (batch tested with PRs 54460, 55258, 54858, 55506, 55510). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. Fix CRI fluentd config. This should fix the cri-containerd stackdriver test failure: ``` Cluster level logging implemented by Stackdriver should ingest logs ``` I copied the pattern from a comment previously. However, it doesn't actually work properly. `\b` only matches word boundary, and seems to match the boundary of previous word in our case. That's why we get the log with a leading space: ``` Nov 10 18:39:11.661: INFO: Unexpected error occurred: log entry ingested incorrectly, got --> <--I0101 00:00:00.000000 1 main.go:1] Text, want Text ``` @kubernetes/sig-node-bugs @kubernetes/sig-instrumentation-bugs Signed-off-by: Lantao Liu <lantaol@google.com> ```release-note none ``` |
||
|---|---|---|
| .. | ||
| es-image | ||
| fluentd-es-image | ||
| es-service.yaml | ||
| es-statefulset.yaml | ||
| fluentd-es-configmap.yaml | ||
| fluentd-es-ds.yaml | ||
| kibana-deployment.yaml | ||
| kibana-service.yaml | ||
| OWNERS | ||
| README.md | ||
Elasticsearch Add-On
This add-on consists of a combination of Elasticsearch, Fluentd and Kibana. Elasticsearch is a search engine that is responsible for storing our logs and allowing for them to be queried. Fluentd sends log messages from Kubernetes to Elasticsearch, whereas Kibana is a graphical interface for viewing and querying the logs stored in Elasticsearch.
Note: this addon should not be used as-is in production. This is an example and you should treat is as such. Please see at least the Security and the Storage sections for more information.
Elasticsearch
Elasticsearch is deployed as a StatefulSet, which is like a Deployment, but allows for maintaining state on storage volumes.
Security
Elasticsearch has capabilities to enable authorization using
X-Pack plugin. See configuration parameter xpack.security.enabled
in Elasticsearch and Kibana configurations. It can also be set via
XPACK_SECURITY_ENABLED env variable. After enabling the feature,
follow official documentation to set up credentials in
Elasticsearch and Kibana. Don't forget to propagate those credentials also to
Fluentd in its configuration, using for example
environment variables. You can utilize ConfigMaps
and Secrets to store credentials in the Kubernetes apiserver.
Initialization
The Elasticsearch Statefulset manifest specifies that there shall be an
init container executing before Elasticsearch containers
themselves, in order to ensure that the kernel state variable
vm.max_map_count is at least 262144, since this is a requirement of
Elasticsearch. You may remove the init container if you know that your host
OS meets this requirement.
Storage
The Elasticsearch StatefulSet will use the EmptyDir volume to store data. EmptyDir is erased when the pod terminates, here it is used only for testing purposes. Important: please change the storage to persistent volume claim before actually using this StatefulSet in your setup!
Fluentd
Fluentd is deployed as a DaemonSet which spawns a pod on each node that reads logs, generated by kubelet, container runtime and containers and sends them to Elasticsearch.
Note: in order for Fluentd to work, every Kubernetes node must be labeled
with beta.kubernetes.io/fluentd-ds-ready=true, as otherwise the Fluentd
DaemonSet will ignore them.
Learn more in the official Kubernetes documentation.
Known problems
Since Fluentd talks to the Elasticsearch service inside the cluster, instances on masters won't work, because masters have no kube-proxy. Don't mark masters with a label mentioned in the previous paragraph or add a taint on them to avoid Fluentd pods scheduling there.