fix logrotate config (again)

we need to add the dateformat option so that the logrotate
can create unique logfiles for each rotation. Without this,
we logrotation is skipped with message like (generated in
verbose mode of logrotate):

rotating log /var/log/rotate-test.log, log->rotateCount is 5
dateext suffix '-20160718'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
destination /var/log/rotate-test2.log-20160718.gz already exists, skipping rotation

Tested as follows:

  # config in '/etc/logrotate.d/rotate-test':
  /var/log/rotate-test.log {
    rotate 5
    copytruncate
    missingok
    notifempty
    compress
    maxsize 100M
    daily
    dateext
    dateformat -%Y%m%d-%s
    create 0644 root root
  }

  # create 150Mb of /var/log/rotate-test.log
  $ dd if=/dev/zero of=/var/log/rotate-test.log bs=1048576 count=150 conv=notrunc oflag=append

  # run logrotate
  $ /usr/sbin/logrotate -v /etc/logrotate.conf
  ...
  rotating pattern: /var/log/rotate-test.log  after 1 days (5 rotations)
  empty log files are not rotated, log files >= 104857600 are rotated earlier, old logs are removed
  considering log /var/log/rotate-test.log
    log needs rotating
  rotating log /var/log/rotate-test.log, log->rotateCount is 5
  Converted ' -%Y%m%d-%s' -> '-%Y%m%d-%s'
  dateext suffix '-20160718-1468875268'
  glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
  copying /var/log/rotate-test.log to /var/log/rotate-test.log-20160718-1468875268
  truncating /var/log/rotate-test.log
  compressing log with: /bin/gzip

  Repeating 'dd' and 'logrotate' commands now generate logfiles correctly.

cluster/gce/gci/configure-helper.sh

@@ -77,12 +77,21 @@ function setup-logrotate() {
     compress
     maxsize 10M
     daily
+    dateext
+    dateformat -%Y%m%d-%s
     create 0644 root root
 }
 EOF
 
   # Configuration for k8s services that redirect logs to /var/log/<service>.log
-  # files.
+  # files. Whenever logrotate is ran, this config will:
+  # * rotate the log file if its size is > 100Mb OR if one day has elapsed
+  # * save rotated logs into a gzipped timestamped backup
+  # * log file timestamp (controlled by 'dateformat') includes seconds too. this
+  #   ensures that logrotate can generate unique logfiles during each rotation
+  #   (otherwise it skips rotation if 'maxsize' is reached multiple times in a
+  #   day).
+  # * keep only 5 old (rotated) logs, and will discard older logs.
   local logrotate_files=( "kube-scheduler" "kube-proxy" "kube-apiserver" "kube-controller-manager" "kube-addons" )
   for file in "${logrotate_files[@]}" ; do
     cat > /etc/logrotate.d/${file} <<EOF
@@ -92,8 +101,10 @@ EOF
     missingok
     notifempty
     compress
-    size 100M
+    maxsize 100M
     daily
+    dateext
+    dateformat -%Y%m%d-%s
     create 0644 root root
 }
 EOF

cluster/saltbase/salt/logrotate/conf

@@ -4,8 +4,10 @@
     missingok
     notifempty
     compress
-    size 100M
+    maxsize 100M
     daily
+    dateext
+    dateformat -%Y%m%d-%s
     create 0644 root root
 }