Commit Graph

71 Commits

Author SHA1 Message Date
zhengjiajin
2ab0c65506 Update docs for user-guide 2017-06-27 12:21:49 +08:00
Mik Vyatskov
2e7eea594c Remove fluentd-gcp image sources 2017-03-01 01:47:01 +01:00
Tim St. Clair
4b4c3e4944
Cleanup fluentd-gcp image, rebase on debian-base 2017-02-24 13:45:12 -08:00
Kubernetes Submit Queue
bb5fdff58b Merge pull request #41567 from Crassirostris/fluentd-gcp-monitoring
Automatic merge from submit-queue (batch tested with PRs 39855, 41433, 41567, 41887, 41652)

Add fluentd monitoring to fluentd-gcp image

Right now we are not able to monitor the state of fluentd in cluster, which may result in logging subsystem quietly failing. This PR tries to address that problem by introducing the fluentd container monitoring:

* fluentd internal metrics, like number of buffers and number of data in buffers
* `logging_line_count`, number of lines, read by fluentd from application containers' logs
    * Has `tag` label, corresponding to the fluentd tag of the entry
* `logging_entry_count`, number of entries, emitted to the output plugin
    * With label `component` set to `container`, generated by application containers
    * With label `component` set to `system`, generated by system components like kubelet, docker, scheduler, etc.
    * Has `tag` label, corresponding to the fluentd tag of the entry

CC @fabxc @igorpeshansky @edsiper
2017-02-23 09:36:33 -08:00
Mik Vyatskov
8d2d91070a Add fluentd monitoring to fluentd-gcp image 2017-02-16 19:04:32 +01:00
Mik Vyatskov
e8de31623f Bump fluentd-gcp google_cloud plugin version 2017-02-16 16:49:16 +01:00
Anthony Yeh
257a8745e3 fluentd-gcp: Add kube-apiserver-audit.log. 2017-02-14 14:23:36 -08:00
Mik Vyatskov
a1ec542d7c Fix copying systemd libraries upon fluentd-gcp startup 2017-02-14 15:41:15 +01:00
Mik Vyatskov
7b194d496f Fix fluentd-gcp configuration to handle different timezones on the node 2017-01-24 11:53:15 +01:00
thomasschickinger
42fbf93fb0 Add rule for detecting exceptions to fluentd config for GKE logging.
Bump version of gcp-fluentd container to 1.34
2017-01-19 15:51:47 +01:00
Mik Vyatskov
83df5b8495 Remove library copying from fluentd image 2017-01-17 15:00:48 +01:00
Jeff Grafton
1c2ea28080 Update images that use ubuntu-slim base image to :0.6 2017-01-11 15:07:04 -08:00
Jeff Grafton
19aafd291c Always --pull in docker build to ensure recent base images 2017-01-10 16:21:05 -08:00
Mik Vyatskov
57ec7b77fd Fix fluentd-gcp image config by avoiding processing its own logs 2017-01-09 10:05:33 +01:00
Kubernetes Submit Queue
2d15499984 Merge pull request #39151 from Crassirostris/fluentd-gcp-default-format
Automatic merge from submit-queue

Try parse golang logs by default

Glog by default logs to stderr, so Stackdriver Logging shows them all as errors. This PR makes fluentd try to parse messages using glog format and if succeeded, set timestamp and severity accordingly.

CC @piosz @fgrzadkowski
2017-01-03 05:50:33 -08:00
Mik Vyatskov
5105102940 Try parse golang logs by default 2016-12-20 17:31:13 +01:00
Lucas Käldström
3c5b5f5963 Remove all MAINTAINER statements in the codebase as they aren't very useful and now deprecated 2016-12-17 20:34:10 +02:00
Mik Vyatskov
e52c3e77e2 Avoid exporting fluentd-gcp own logs 2016-12-14 14:43:05 +01:00
Mik Vyatskov
6215089796 Unify fluentd-gcp configurations 2016-11-30 14:47:24 +01:00
Mik Vyatskov
ba19a27974 Bump version of the fluentd-gcp image 2016-11-22 11:12:28 +01:00
Mik Vyatskov
b5b47620f5 Add kube-proxy logs to fluentd configs 2016-11-21 15:43:28 +01:00
Mik Vyatskov
aa31f7bc02 Bump fluentd-gcp image version 2016-11-17 21:21:10 +01:00
Mik Vyatskov
cf845eda7c Remove fixed docker daemon log time format in fluentd configs 2016-11-17 15:52:07 +01:00
Mik Vyatskov
279e20ed13 Fix flunetd-gcp image Dockerfile 2016-11-08 15:14:09 +01:00
Mik Vyatskov
d478307106 Fix config file names inside fluentd-gcp image 2016-11-07 20:31:12 +01:00
Kubernetes Submit Queue
04a81cdd3e Merge pull request #36363 from Crassirostris/fluentd-gcp-image-build-fix
Automatic merge from submit-queue

Fix fluentd-gcp Dockerfile to reduce image size

Change reduces image size by 150MB.

@piosz
2016-11-07 07:56:11 -08:00
Mik Vyatskov
d3465e5b8c Add rescheduler logs to the fluentd-gcp configuration 2016-11-07 15:10:52 +01:00
Mik Vyatskov
220168c9aa Fix fluentd-gcp Dockerfile to reduce image size 2016-11-07 15:01:38 +01:00
Mik Vyatskov
b4c7dfa674 Switched from google-fluentd to configuring fluentd from scratch 2016-11-02 12:08:36 +01:00
Mik Vyatskov
24361fce06 Update fluentd-gcp configuration 2016-11-02 12:06:01 +01:00
Jess Frazelle
cde5559d09
Update gcloud docker commands to use gcloud docker -- ARGS
We can then avoid the following warning:
```
WARNING: The '--' argument must be specified between gcloud specific args on the left and DOCKER_ARGS on the right. IMPORTANT: previously, commands allowed the omission of the --, and unparsed arguments were treated as implementation args. This usage is being deprecated and will be removed in March 2017.
This will be strictly enforced in March 2017. Use 'gcloud beta docker' to see new behavior.
```

Signed-off-by: Jess Frazelle <acidburn@google.com>
2016-10-10 13:42:34 -07:00
Manuel de Brito Fontes
e2c50152dd Remove kubernetes metadata plugin 2016-08-23 08:21:46 -03:00
Manuel de Brito Fontes
5753a3196c Reduce the size of fluentd images 2016-08-23 08:21:45 -03:00
Rick Buskens
77bda6fd16 Updated configuration to spawn multiple threads for processing 2016-08-01 15:08:11 -04:00
David McMahon
ef0c9f0c5b Remove "All rights reserved" from all the headers. 2016-06-29 17:47:36 -07:00
Alex Robinson
19bf9d0624 Support journal logs in fluentd-gcp.
Only run the systemd-journal plugin when on a platform that requests it.
The plugin crashes the fluentd process if the journal isn't present, so
it can't just be run blindly in all configurations.
2016-06-24 00:30:32 +00:00
Alex Robinson
bb51c94172 Add collection of the new glbc and cluster-autoscaler logs 2016-06-03 15:50:52 +00:00
Alex Robinson
3216204668 Update to the latest version of google-fluentd.
It includes some performance improvements for parsing JSON (which is
very important for us, since all Docker logs are JSON) as well as a
couple new settings, like forcing of a flush of multiline logs after a
time period rather than having to wait until a new log is seen before
feeling confident flushing the previous one.
2016-03-05 00:18:31 +00:00
Alex Robinson
29eb6020fb Merge pull request #22074 from a-robinson/images
Update the logging images to parse sub-second docker timestamps.
2016-03-04 09:04:33 -08:00
Alex Robinson
feea376b5c Update the logging images to parse sub-second docker timestamps. 2016-03-03 10:21:30 -08:00
Alex Robinson
f8965a6f41 Fix parsing of kube logs to handle logs split across lines.
I didn't expect glog to split single log statements onto multiple lines,
but apparently it does if they're long enough. This groups them back
together appropriately.
2016-02-26 22:58:46 +00:00
Christopher Eck
c34ecf7100 Enable fluentd to parse the sub-second portion of timestamps coming from docker logs. 2016-02-18 11:07:27 -08:00
Alex Robinson
e5c5f76402 Update the default fluentd images to parse system logs. 2016-02-04 23:29:27 +00:00
Alex Robinson
592da85a81 Parse system logs into structured messages in fluentd.
This allows you to filter based on components of the log,
like their severity, in the developers console or elasticsearch.
2016-02-04 08:14:59 +00:00
Paul Morie
b672785d72 Add boilerplate checks for Dockerfiles 2016-02-03 18:35:26 -05:00
Paul Morie
05bd107301 Add boilerplate checks for Makefiles 2016-02-03 18:35:26 -05:00
Alex Robinson
6acf297231 Update the fluentd versions to include fixes for #19405. 2016-01-27 00:30:53 +00:00
Alex Robinson
ac13e851e2 Don't let fluentd pipe its own logs directly back into itself. 2016-01-27 00:30:53 +00:00
Alex Robinson
95a4ea3fc8 Increase the fluentd buffer chunk size to improve write throughput.
Also reduce the max wait between retries, 30 seconds should be more than
enough backoff.
2016-01-27 00:30:50 +00:00
Alex Robinson
f0ebe4066f Update fluentd-gcp to include all recent improvements that are in the
new google-fluentd 1.5.4-1 build.
2015-10-29 11:54:07 +00:00