diff --git a/docs/getting-started-guides/es-browser.png b/docs/getting-started-guides/es-browser.png new file mode 100644 index 00000000000..f556fa8c561 Binary files /dev/null and b/docs/getting-started-guides/es-browser.png differ diff --git a/docs/getting-started-guides/kibana-logs.png b/docs/getting-started-guides/kibana-logs.png new file mode 100644 index 00000000000..15b2f6759b3 Binary files /dev/null and b/docs/getting-started-guides/kibana-logs.png differ diff --git a/docs/getting-started-guides/logging-elasticsearch.md b/docs/getting-started-guides/logging-elasticsearch.md new file mode 100644 index 00000000000..df35894543b --- /dev/null +++ b/docs/getting-started-guides/logging-elasticsearch.md @@ -0,0 +1,222 @@ +# Cluster Level Logging with Elasticsearch and Kibana + +On the GCE platform the default cluster level logging support targets +[Google Cloud Logging](https://cloud.google.com/logging/docs/) as described at the [Logging](logging.md) getting +started page. Here we describe how to set up a cluster to ingest logs into Elasticsearch and view them using Kibana as an +alternative to Google Cloud Logging. + +To use Elasticsearch and Kibana for cluster logging you should set the following environment variable as shown below: +``` +KUBE_LOGGING_DESTINATION=elasticsearch +``` + +You should also ensure that `KUBE_ENABLE_NODE_LOGGING=true` (which is the default for the GCE platform). + +Now when you create a cluster a message will indicate that the Fluentd node-level log collectors +will target Elasticsearch: + +``` +$ cluster/kube-up.sh +... +Project: kubernetes-satnam +Zone: us-central1-b +... calling kube-up +Project: kubernetes-satnam +Zone: us-central1-b ++++ Staging server tars to Google Storage: gs://kubernetes-staging-e6d0e81793/devel ++++ kubernetes-server-linux-amd64.tar.gz uploaded (sha1 = 6987c098277871b6d69623141276924ab687f89d) ++++ kubernetes-salt.tar.gz uploaded (sha1 = bdfc83ed6b60fa9e3bff9004b542cfc643464cd0) +Looking for already existing resources +Starting master and configuring firewalls +Created [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/zones/us-central1-b/disks/kubernetes-master-pd]. +NAME ZONE SIZE_GB TYPE STATUS +kubernetes-master-pd us-central1-b 20 pd-ssd READY +Created [https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/regions/us-central1/addresses/kubernetes-master-ip]. ++++ Logging using Fluentd to elasticsearch + +``` + +The node level Fluentd collector pods and the Elasticsearech pods used to ingest cluster logs and the pod for the Kibana +viewer should be running soon after the cluster comes to life. + +``` +$ kubectl get pods +NAME READY REASON RESTARTS AGE +elasticsearch-logging-v1-78nog 1/1 Running 0 2h +elasticsearch-logging-v1-nj2nb 1/1 Running 0 2h +fluentd-elasticsearch-kubernetes-minion-5oq0 1/1 Running 0 2h +fluentd-elasticsearch-kubernetes-minion-6896 1/1 Running 0 2h +fluentd-elasticsearch-kubernetes-minion-l1ds 1/1 Running 0 2h +fluentd-elasticsearch-kubernetes-minion-lz9j 1/1 Running 0 2h +kibana-logging-v1-bhpo8 1/1 Running 0 2h +kube-dns-v3-7r1l9 3/3 Running 0 2h +monitoring-heapster-v4-yl332 1/1 Running 1 2h +monitoring-influx-grafana-v1-o79xf 2/2 Running 0 2h + +``` + +Here we see that for a four node cluster there is a `fluent-elasticsearch` pod running which gathers +the Docker container logs and sends them to Elasticsearch. The Fluentd collector communicates to +a Kubernetes service that maps requests to specific Elasticsearch pods. Similarly, Kibana can also be +accessed via a Kubernetes service definition. + + +``` +$ kubectl get services +NAME LABELS SELECTOR IP(S) PORT(S) +elasticsearch-logging k8s-app=elasticsearch-logging,kubernetes.io/cluster-service=true,kubernetes.io/name=Elasticsearch k8s-app=elasticsearch-logging 10.0.222.57 9200/TCP +kibana-logging k8s-app=kibana-logging,kubernetes.io/cluster-service=true,kubernetes.io/name=Kibana k8s-app=kibana-logging 10.0.193.226 5601/TCP +kube-dns k8s-app=kube-dns,kubernetes.io/cluster-service=true,kubernetes.io/name=KubeDNS k8s-app=kube-dns 10.0.0.10 53/UDP + 53/TCP +kubernetes component=apiserver,provider=kubernetes 10.0.0.1 443/TCP +monitoring-grafana kubernetes.io/cluster-service=true,kubernetes.io/name=Grafana k8s-app=influxGrafana 10.0.167.139 80/TCP +monitoring-heapster kubernetes.io/cluster-service=true,kubernetes.io/name=Heapster k8s-app=heapster 10.0.208.221 80/TCP +monitoring-influxdb kubernetes.io/cluster-service=true,kubernetes.io/name=InfluxDB k8s-app=influxGrafana 10.0.188.57 8083/TCP +``` + +By default two Elasticsearch replicas are created and one Kibana replica is created. + +``` +$ kubectl get rc +CONTROLLER CONTAINER(S) IMAGE(S) SELECTOR REPLICAS +elasticsearch-logging-v1 elasticsearch-logging gcr.io/google_containers/elasticsearch:1.4 k8s-app=elasticsearch-logging,version=v1 2 +kibana-logging-v1 kibana-logging gcr.io/google_containers/kibana:1.3 k8s-app=kibana-logging,version=v1 1 +kube-dns-v3 etcd gcr.io/google_containers/etcd:2.0.9 k8s-app=kube-dns,version=v3 1 + kube2sky gcr.io/google_containers/kube2sky:1.9 + skydns gcr.io/google_containers/skydns:2015-03-11-001 +monitoring-heapster-v4 heapster gcr.io/google_containers/heapster:v0.14.3 k8s-app=heapster,version=v4 1 +monitoring-influx-grafana-v1 influxdb gcr.io/google_containers/heapster_influxdb:v0.3 k8s-app=influxGrafana,version=v1 1 + grafana gcr.io/google_containers/heapster_grafana:v0.7 +``` + +The Elasticsearch and Kibana services are not directly exposed via a publicly reachable IP address. Instead, +they can be accessed via the service proxy running at the master. The URLs for accessing Elasticsearch +and Kibana via the service proxy can be found using the `kubectl cluster-info` command. + +``` +$ kubectl cluster-info +Kubernetes master is running at https://146.148.94.154 +Elasticsearch is running at https://146.148.94.154/api/v1/proxy/namespaces/default/services/elasticsearch-logging +Kibana is running at https://146.148.94.154/api/v1/proxy/namespaces/default/services/kibana-logging +KubeDNS is running at https://146.148.94.154/api/v1/proxy/namespaces/default/services/kube-dns +Grafana is running at https://146.148.94.154/api/v1/proxy/namespaces/default/services/monitoring-grafana +Heapster is running at https://146.148.94.154/api/v1/proxy/namespaces/default/services/monitoring-heapster +InfluxDB is running at https://146.148.94.154/api/v1/proxy/namespaces/default/services/monitoring-influxdb + +``` + +Before accessing the logs ingested into Elasticsearch using a browser and the service proxy URL we need to find out +the `admin` password for the cluster using `kubectl config view`. + +``` +$ kubectl config view +... +- name: kubernetes-satnam_kubernetes-basic-auth + user: + password: 7GlspJ9Q43OnGIJO + username: admin +... +``` + +The first time you try to access the cluster from a browser a dialog box appears asking for the username and password. +Use the username `admin` and provide the basic auth password reported by `kubectl config view` for the +cluster you are trying to connect to. Connecting to the Elasticsearch URL should then give the +status page for Elasticsearch. + +![Elasticsearch Status](es-browser.png) + +You can now type Elasticsearch queries directly into the browser. Alternatively you can query Elasticsearch +from your local machine using `curl` but first you need to know what your bearer token is: +``` +$ kubectl config view --minify +apiVersion: v1 +clusters: +- cluster: + certificate-authority-data: REDACTED + server: https://146.148.94.154 + name: kubernetes-satnam_kubernetes +contexts: +- context: + cluster: kubernetes-satnam_kubernetes + user: kubernetes-satnam_kubernetes + name: kubernetes-satnam_kubernetes +current-context: kubernetes-satnam_kubernetes +kind: Config +preferences: {} +users: +- name: kubernetes-satnam_kubernetes + user: + client-certificate-data: REDACTED + client-key-data: REDACTED + token: JsUe2Z3cXqa17UQqQ8qWGGf4nOSLwSnp +``` + +Now you can can issue requests to Elasticsearch: +``` +$ curl --header "Authorization: Bearer JsUe2Z3cXqa17UQqQ8qWGGf4nOSLwSnp" --insecure https://146.148.94.154/api/v1/proxy/namespaces/default/services/elasticsearch-logging/ +{ + "status" : 200, + "name" : "Vance Astrovik", + "cluster_name" : "kubernetes-logging", + "version" : { + "number" : "1.5.2", + "build_hash" : "62ff9868b4c8a0c45860bebb259e21980778ab1c", + "build_timestamp" : "2015-04-27T09:21:06Z", + "build_snapshot" : false, + "lucene_version" : "4.10.4" + }, + "tagline" : "You Know, for Search" +} + +``` +Note that you need the trailing slash at the end of the service proxy URL. Here is an example of a search: +``` +$ curl --header "Authorization: Bearer JsUe2Z3cXqa17UQqQ8qWGGf4nOSLwSnp" --insecure https://146.148.94.154/api/v1/proxy/namespaces/default/services/elasticsearch-logging/_search?pretty=true +{ + "took" : 7, + "timed_out" : false, + "_shards" : { + "total" : 6, + "successful" : 6, + "failed" : 0 + }, + "hits" : { + "total" : 123711, + "max_score" : 1.0, + "hits" : [ { + "_index" : ".kibana", + "_type" : "config", + "_id" : "4.0.2", + "_score" : 1.0, + "_source":{"buildNum":6004,"defaultIndex":"logstash-*"} + }, { +... + "_index" : "logstash-2015.06.22", + "_type" : "fluentd", + "_id" : "AU4c_GvFZL5p_gZ8dxtx", + "_score" : 1.0, + "_source":{"log":"synthetic-logger-10lps-pod: 31: 2015-06-22 20:35:33.597918073+00:00\n","stream":"stdout","tag":"kubernetes.synthetic-logger-10lps-pod_default_synth-lgr","@timestamp":"2015-06-22T20:35:33+00:00"} + }, { + "_index" : "logstash-2015.06.22", + "_type" : "fluentd", + "_id" : "AU4c_GvFZL5p_gZ8dxt2", + "_score" : 1.0, + "_source":{"log":"synthetic-logger-10lps-pod: 36: 2015-06-22 20:35:34.108780133+00:00\n","stream":"stdout","tag":"kubernetes.synthetic-logger-10lps-pod_default_synth-lgr","@timestamp":"2015-06-22T20:35:34+00:00"} + } ] + } +} +``` + +The Elasticsearch website contains information about [URI search queries](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-uri-request.html) which can be used to extract the required logs. + +Alternatively you can view the ingested logs using Kibana. The first time you visit the Kibana URL you will be +presented with a page that asks you to configure your view of the ingested logs. Select the option for +timeseries values and select `@timestamp`. On the following page select the `Discover` tab and then you +should be able to see the ingested logs. You can set the refresh interval to 5 seconds to have the logs +regulary refreshed. Here is a typical view of ingested logs from the Kibana viewer. + +![Kibana logs](kibana-logs.png) + + + +[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/getting-started-guides/logging-elasticsearch.md?pixel)]()