github/kubernetes
4
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
49,112 Commits 1 Branch 31 Tags 1,019 MiB
cb6be1eb06
Commit Graph

158 Commits

Author SHA1 Message Date
Mik Vyatskov
3b6353bdc8 Fixed kibana image and controller to work through proxy 2016-11-02 22:06:22 +01:00
Matt Condon
00f6592d9e fix: elasticsearch template mapping to parse kubernetes.labels 2016-10-28 16:29:34 -04:00
Jan Chaloupka
4fde09d308 Replace client with clientset in code 2016-10-23 22:00:35 +02:00
Kubernetes Submit Queue
35943d6ad6 Merge pull request #34969 from Crassirostris/es-kibana-usage-update 
Automatic merge from submit-queue

Update elasticsearch and kibana usage

```release-note
Updated default Elasticsearch and Kibana used for elasticsearch logging destination to versions 2.4.1 and 4.6.1 respectively.
```

Updated controllers for elasticsearch and kibana to use newer versions of images. Fixed e2e test because of elasticsearch backward incompatible API changes.

Fixed out of sync elasticsearch controller for coreos.

@piosz
 2016-10-20 12:26:02 -07:00
Eduardo Silva
5e321af2ed cluser/addons: fluentd-elasticsearch: preload Jemalloc for td-agent 
The current DockerFile build an image using td-agent package but it let
the service run with the default memory allocator provided by glibc.

In high load environments, is highly required to use a customized memory
allocator such as Jemalloc. Otherwise the service will face a high memory
fragmentation ending up in 'high memory' usage from a monitoring perspective.

td-agent package by default install Jemalloc and set the LD_PRELOAD
environment variable through it init script, but since the service is
launched through Docker the env variable needs to be set manually.

After this patch, when running td-agent container image now is possible
to see that Jemalloc is used:

  root@monotop:/proc/18810# cat maps |grep jemall
  7f251eddd000-7f251ee1b000 ... /opt/td-agent/embedded/lib/libjemalloc.so.2
  7f251ee1b000-7f251f01b000 ... /opt/td-agent/embedded/lib/libjemalloc.so.2
  7f251f01b000-7f251f01d000 ... /opt/td-agent/embedded/lib/libjemalloc.so.2
  7f251f01d000-7f251f01e000 ... /opt/td-agent/embedded/lib/libjemalloc.so.2

For a reference about the memory usage difference between malloc v/s jemalloc
please refer to the following chart:

  https://goo.gl/dVYTmw

Signed-off-by: Eduardo Silva <eduardo@treasure-data.com>
 2016-10-20 10:50:58 -06:00
Mik Vyatskov
9832ae1259 Update elasticsearch and kibana usage 2016-10-17 21:04:25 +02:00
Mik Vyatskov
7a582fcae5 Fixed kibana image 2016-10-17 17:45:59 +02:00
Kubernetes Submit Queue
fd1fd2a14c Merge pull request #34562 from Crassirostris/es-kibana-update 
Automatic merge from submit-queue

Elasticsearch and Kibana update

```release-note
Updated Elasticsearch image from version 1.5.1 to version 2.4.1. Updated Kibana image from version 4.0.2 to version 4.6.1.
```

Updated es and kibana images. Made image versions match es/kibana versions they contain.

ref #19149
 2016-10-17 03:04:33 -07:00
Mik Vyatskov
5b79b65bd3 Update elasticsearch and kibana images 2016-10-16 16:01:38 +02:00
Jess Frazelle
cde5559d09
Update gcloud docker commands to use gcloud docker -- ARGS 
We can then avoid the following warning:
```
WARNING: The '--' argument must be specified between gcloud specific args on the left and DOCKER_ARGS on the right. IMPORTANT: previously, commands allowed the omission of the --, and unparsed arguments were treated as implementation args. This usage is being deprecated and will be removed in March 2017.
This will be strictly enforced in March 2017. Use 'gcloud beta docker' to see new behavior.
```

Signed-off-by: Jess Frazelle <acidburn@google.com>
 2016-10-10 13:42:34 -07:00
Mik Vyatskov
3fbde5ecfb Fixed elasticsearch cluster logging e2e test on GCI 2016-09-21 13:55:43 +02:00
Manuel de Brito Fontes
e2c50152dd Remove kubernetes metadata plugin 2016-08-23 08:21:46 -03:00
Manuel de Brito Fontes
5753a3196c Reduce the size of fluentd images 2016-08-23 08:21:45 -03:00
haoyuan
a1f20cbdf4 fix typo 2016-08-08 19:33:51 +08:00
Rick Buskens
77bda6fd16 Updated configuration to spawn multiple threads for processing 2016-08-01 15:08:11 -04:00
David McMahon
ef0c9f0c5b Remove "All rights reserved" from all the headers. 2016-06-29 17:47:36 -07:00
pprokop
fa91b3e461 Adding option to deploy fluentd-elasticsearch in diffrent namespace then kube-system 2016-06-27 18:15:41 +02:00
Alex Robinson
bb51c94172 Add collection of the new glbc and cluster-autoscaler logs 2016-06-03 15:50:52 +00:00
Alex Robinson
8579509293 Rebuild elasticsearch image to include changes since 1.2 2016-06-01 20:48:20 +00:00
keontang
1520a01d97 Fixed #25127 2016-05-16 09:03:00 +08:00
Mike Zupan
dc643d2592 Fixing a simple typo 2016-03-06 23:49:35 -08:00
Alex Robinson
29eb6020fb Merge pull request #22074 from a-robinson/images 
Update the logging images to parse sub-second docker timestamps.
 2016-03-04 09:04:33 -08:00
Alex Robinson
feea376b5c Update the logging images to parse sub-second docker timestamps. 2016-03-03 10:21:30 -08:00
Alex Robinson
f8965a6f41 Fix parsing of kube logs to handle logs split across lines. 
I didn't expect glog to split single log statements onto multiple lines,
but apparently it does if they're long enough. This groups them back
together appropriately.
 2016-02-26 22:58:46 +00:00
Christopher Eck
c34ecf7100 Enable fluentd to parse the sub-second portion of timestamps coming from docker logs. 2016-02-18 11:07:27 -08:00
Alex Robinson
e5c5f76402 Update the default fluentd images to parse system logs. 2016-02-04 23:29:27 +00:00
Alex Robinson
592da85a81 Parse system logs into structured messages in fluentd. 
This allows you to filter based on components of the log,
like their severity, in the developers console or elasticsearch.
 2016-02-04 08:14:59 +00:00
Paul Morie
b672785d72 Add boilerplate checks for Dockerfiles 2016-02-03 18:35:26 -05:00
Paul Morie
05bd107301 Add boilerplate checks for Makefiles 2016-02-03 18:35:26 -05:00
Alex Robinson
6acf297231 Update the fluentd versions to include fixes for #19405. 2016-01-27 00:30:53 +00:00
Alex Robinson
ac13e851e2 Don't let fluentd pipe its own logs directly back into itself. 2016-01-27 00:30:53 +00:00
Alex Robinson
95a4ea3fc8 Increase the fluentd buffer chunk size to improve write throughput. 
Also reduce the max wait between retries, 30 seconds should be more than
enough backoff.
 2016-01-27 00:30:50 +00:00
gmarek
2d5c429f9d Add resource requests to addons 2015-12-17 10:13:15 +01:00
Minhan Xia
b4c72e56ce enforce node quorum to prevent es split brain problem 2015-12-14 18:16:04 -08:00
Matt Moore
b750d1dddc Drop the beta for GCR v2 images. 
beta.gcr.io is no longer needed to pull through v2.
 2015-11-14 12:19:02 -08:00
Matt Moore
eeb4eeb17c Move pause and fluentd-elasticsearch to GCR v2. 
This scopes down the initially ambitious PR:
https://github.com/kubernetes/kubernetes/pull/14960 to replace just
`pause` and `fluentd-elasticsearch` to come through `beta.gcr.io`.

The v2 versions have been pushed under new tags, `pause:2.0` and
`fluentd-elastisearch:1.12`.

NOTE: `beta.gcr.io` will still serve images using v1 until they are repushed with v2.  Pulls through `gcr.io` will still work after pushing through `beta.gcr.io`, but will be served over v1 (via compat logic).
 2015-10-06 16:39:07 -07:00
Tamer Tas
cb5f921df1 Add newline to the end of bash scripts 2015-09-18 19:35:05 +03:00
Eric Paris
04fc8ae3dd s|github.com/GoogleCloudPlatform/kubernetes|github.com/kubernetes/kubernetes| 2015-09-03 10:10:11 -04:00
Jimmi Dyson
e07da97447 Switch to fluentd kubernetes metadata plugin to enrich logs with more searchable kubernetes metadata 2015-09-02 10:33:15 +01:00
Alex Robinson
d02dc64036 Let fluentd-es log by not redirecting to a file within the container 
and only using -q (warning logs only) rather than -qq (error logs only).
 2015-08-27 21:41:44 +00:00
Alex Robinson
f6f8cb07ae Transfer maintainer responsibilities for fluentd configs/images. 2015-08-26 21:48:51 +00:00
Satnam Singh
278a3911fd Fetch fluentd td agent securely 2015-08-19 13:34:03 -07:00
Kris Rousey
ae6c64d9bb Moving everyone to unversioned client 2015-08-18 10:23:03 -07:00
Alex Robinson
c887ad75b8 Stop collecting monit logs via fluentd. 2015-08-07 00:12:21 +00:00
Mike Danese
17defc7383 run gofmt on everything we touched 2015-08-05 17:52:56 -07:00
Mike Danese
8e33cbfa28 rewrite go imports 2015-08-05 17:30:03 -07:00
Alex Robinson
cc02e50fc7 Collect more of the VM's log files (monit, salt) in fluentd-es. 2015-08-05 00:22:38 +00:00
Alex Robinson
69ab7026af Update fluentd-es to collect etcd logs and remove merge conflict cruft. 2015-07-30 22:21:01 +00:00
Satnam Singh
691c07763c Remove use of v1beta3 from ES logging 2015-07-06 18:47:45 -07:00
Brendan Burns
988aa6fdf6 Move things into a 'kube-system' namespace. 2015-07-06 15:08:23 -07:00
Zach Loafman
a2250d1caf Merge pull request #10597 from a-robinson/fluentd 
Run fluentd on the master to collect the core master logs
 2015-07-04 21:12:47 -07:00
Alex Robinson
4bfbd5f553 Add limits to the fluentd buffer size. 2015-07-03 23:24:08 +00:00
Zach Loafman
1d16be675c Merge pull request #10653 from dchen1107/resource_management 
Set resource limit for addon containers
 2015-07-01 19:35:28 -07:00
Dawn Chen
6b61918b67 Set resource limit for both elasticsearch and kibana containers based on data collected by #10335 2015-07-01 16:01:09 -07:00
nikhiljindal
274792d7bb Stop exposing v1beta3 by default 2015-07-01 14:38:02 -07:00
Alex Robinson
8855371763 Configure fluentd to collect additional log files, including master log files. 
This also merges the output blocks together because there's no compelling
reason to have a separate output block for each input.
 2015-07-01 20:07:47 +00:00
Satnam Singh
4e58e001da Change port names to avoid validation failure 2015-06-30 10:56:20 -07:00
Satnam Singh
cb9f7d17f5 Add readiness probes to Elasticsearch and Kibana logging 2015-06-22 12:40:15 -07:00
Tim St. Clair
f5830edbbc s/gcloud preview docker/gcloud docker/ 2015-06-18 12:27:08 -07:00
Satnam Singh
ce49c32d6a Add readiness probe to Elasticsearch and Kibana logging pods 2015-06-18 10:46:05 -07:00
Daniel Smith
40eb1599dd fix elasticsearch 2015-06-03 15:41:09 -07:00
Victor Marmol
7dde223c2a Merge pull request #8458 from satnam6502/elasticsearch 
Give explicit version number to Elasticsearch logging RC
 2015-05-18 16:27:08 -07:00
Satnam Singh
3337cea8bb Give explicit version number to Kibana RC 2015-05-18 16:04:13 -07:00
Victor Marmol
e83a4e3fd8 Merge pull request #8450 from satnam6502/kibana 
Add version label to Kibana controller and service
 2015-05-18 15:56:27 -07:00
Satnam Singh
83cd97a2e7 Give explicit version number to Elasticsearch logging RC 2015-05-18 15:53:01 -07:00
Satnam Singh
4a0c6b3e0b Add version label to Kibana controller and service 2015-05-18 14:49:12 -07:00
Satnam Singh
c84fd94b11 Update Elasticsearch cluster level service to add version label 2015-05-18 14:43:58 -07:00
Jeff Lowdermilk
553f9f822b Add ga-beacon analytics to gendocs scripts 
hack/run-gendocs.sh puts ga-beacon analytics link into all md files,
hack/verify-gendocs.sh verifies presence of link.
 2015-05-15 18:56:38 -07:00
Satnam Singh
ca121ee98d Revert to using emptyDir for Elasticsearch data 2015-05-13 12:52:56 -07:00
Nikhil Jindal
c2fecff3d5 Merge pull request #8067 from piosz/kubectl 
Added label kubernetes.io/name to cluster services
 2015-05-12 17:09:22 -07:00
Satnam Singh
7cff506c93 Make the Elasticsearch logging pods discover each other 2015-05-12 15:26:09 -07:00
Piotr Szczesniak
12e44ff8a9 Added label kubernetes.io/name to cluster services 
The label specifies service name which is displayed in kubectl cluster-info
 2015-05-11 19:35:53 +02:00
Satnam Singh
fcbe2229df Kibana by itself 
Upgrade to Kibana 4 for cluster logging
 2015-05-08 14:55:52 -07:00
Eric Paris
6b3a6e6b98 Make copyright ownership statement generic 
Instead of saying "Google Inc." (which is not always correct) say "The
Kubernetes Authors", which is generic.
 2015-05-01 17:49:56 -04:00
Satnam Singh
95007d7217 Improve comment in Fluentd to ES logging config file 2015-04-30 13:42:30 -07:00
Satnam Singh
e65f0aafd5 Trim cluster log tags to pod name and container name 2015-04-29 18:24:13 -07:00
Satnam Singh
dc31c58443 Upgrade Elasticsearch to 1.5.2 for cluster logging 2015-04-28 16:20:03 -07:00
Alex Robinson
ada9242155 Remove unused env variable from the fluentd dockerfiles and make their apt-get 
arguments consistent.
 2015-04-27 20:47:38 +00:00
Robert Bailey
a9f8f17299 Fetch the kibana tar.gz over SSL. 2015-04-26 21:57:49 -07:00
Nikhil Jindal
84cb48be11 Merge pull request #7246 from satnam6502/es 
Convert Elasticsearch logging to v1beta3 and de-salt
 2015-04-24 09:21:41 -07:00
Satnam Singh
c9b9e7651e Convert Elasticsearch logging to v1beta and de-salt 2015-04-23 13:06:15 -07:00
Satnam Singh
0abc34245a Update Kibana RC and service to v1beta3 2015-04-23 12:19:17 -07:00
Dawn Chen
f9156c281a Merge pull request #7123 from satnam6502/logging 
Propagate pod and container name for log files
 2015-04-23 10:13:08 -07:00
Satnam Singh
e4e830badc Update Kibana pod to speak to Elasticsearchusing v1beta3 2015-04-22 17:37:47 -07:00
Satnam Singh
2444c1f943 Propagate pod and container name for log files 2015-04-22 15:08:51 -07:00
Alex Robinson
059a8c92bd Move the logging-related directories to where I think they belong. 
1. Move fluentd-gcp to be a core cluster addon, rather than a contrib.
2. Get rid of the synthetic logger under contrib, since the exact same
synthetic logger was also included in the logging-demo.
3. Move the logging-demo to examples, since it's effectively an example.

We should also consider adding on a GCP section to the logging-demo
example :)
 2015-04-17 23:59:50 +00:00
Tim Hockin
c2bacd588d Stop using dockerfile/* images 
As per
http://blog.docker.com/2015/03/updates-available-to-popular-repos-update-your-images/
docker has stopped answering dockerfile/redis and dockerfile/nginx.  Fix all
users in our tree.  Sadly this means a lot of published examples are now broken.
 2015-04-16 12:20:43 -07:00
Satnam Singh
4c0f06ec66 Convert es-image for gcr.io 2015-04-15 16:28:02 -04:00
Satnam Singh
cfb8577324 Update Kibana to v1.2 which paramaterizes location of Elasticsearch 2015-04-03 14:55:00 -07:00
Jon Langemak
35038dc19a Added ES ENV variables so Kibana can be access in other ways 2015-04-02 21:58:10 -05:00
Jeff Lowdermilk
506887515d Rename google-containers to google_containers in image names 2015-04-01 09:16:48 -07:00
Satnam Singh
73a9ab2928 Rename logging image to use google_containers 2015-03-31 16:00:52 -07:00
Jeff Lowdermilk
bec982774b Merge pull request #6038 from satnam6502/fluentd-es-image 
Create a gcr.io image for Flunentd to Elasticsearch node level logging
 2015-03-27 09:05:00 -07:00
Satnam Singh
19e621c951 Create a gcr.io image for Flunentd to Elasticsearch node level logging 2015-03-26 21:07:15 -07:00
Jeff Lowdermilk
2b71857816 Pull cluster addon images from gcr.io 2015-03-26 19:59:27 -07:00
Satnam Singh
985d5c1f95 Add a prefix to the POS file for Fluentd to Elasticsearch 2015-03-17 13:13:03 -07:00
Satnam Singh
3ce0ede78e Modify Fluentd to ES setup to pass flags 2015-03-10 10:11:26 -07:00
Satnam Singh
2486de1287 Change Fluentd config to read from head of Docker container logs 2015-03-09 23:25:52 -07:00
Satnam Singh
72183c064c Use proxy instead of load balancer 2015-03-02 13:44:41 -08:00
Tim Hockin
e0e6741027 Stop using named ports in examples and scripts. 2015-03-01 21:30:45 -08:00