Otdfbt

We have a Java-application, that uses log4j and rotates its own logs daily:

• The currently-used log-file is /var/log/foo/foo.log
  


• It is moved into /var/log/foo/foo.log.YYYY-MM-dd when the day changes

We'd rather not change its configuration...

However, we do want to:

1. Compress the file once it has been renamed


2. Only retain a certain number of them

Though it wouldn't be difficult to write a cron-job to do both, we'd rather stay in the framework of logrotate...

I create the following /etc/logrotate.d/foo:

/var/log/foo/foo.log.* 
 daily
 rotate 2
 compress
 delaycompress
 missingok
 notifempty

but it does not do anything:

reading config file /etc/logrotate.d/foo

Handling 1 logs

rotating pattern: /var/log/foo/foo.log.* after 1 days (2 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/foo/foo.log.2017-06-28
 log does not need rotating
considering log /var/log/foo/foo.log.2017-06-29
 log does not need rotating
considering log /var/log/foo/foo.log.2017-06-30
 log does not need rotating
considering log /var/log/foo/foo.log.2017-07-01
 log does not need rotating
considering log /var/log/foo/foo.log.2017-07-02
 log does not need rotating
considering log /var/log/foo/foo.log.2017-07-03
 log does not need rotating
considering log /var/log/foo/foo.log.2017-07-04
 log does not need rotating

How do I make it compress the files and delete the oldest ones, when their total number exceeds 2?

I really don't think you are going be able to do what you want with logrotate without some external commands. Also, the pattern /var/log/foo/foo.log.* would treat each individual file as if it was a separate log to be rotated, not as a set of rotated files.

man logrotate

Please use wildcards with caution. If you specify *, logrotate will rotate all files, including previously rotated ones.

Anyway, maybe something like this? Just set your rotate condition that will never be reached since log4j is already handling the rotation partly. Then put the rest in a prerotate script?

/var/log/foo/foo.log 
 size 100G # something big that will never be reached/rotated since log4j does this.
 missingok
 notifempty
 # delete the other logs over 7 days old
 prerotate
 find /var/log/foo/ -name 'foo.log.*' -mtime +7 -delete
 nice gzip /var/log/foo/foo.log.*
 endscript

I needed to accomplish this behavior because the app teams wanted to maintain control of log and filename format as well as rotation schedule with log4j. If you read man logrotate you'll notice that all the pre/post/lastaction scripts only execute when at least one log was rotated. Thus, setting something unreachable like size 999G does not work.

Instead, what I did was set the copy and extension options so that:

1. The original log file was never touched (it's managed by log4j)


2. The logs rotated by logrotate have a different extension that I can filter on

/var/log/<service>/<service>.log 
 copy # don't touch the orig. log file, it's managed by log4j
 extension .log # End rotated files in ".log" so we can find and delete them
 rotate 1
 hourly
 missingok

 prerotate
 find /var/log/<service>/ -iname "*service.log.*" ! -iname "*.gz" -exec gzip ; && 
 aws s3 sync /var/log/<service>/ s3://<bucket>/<environment>/<ec2_id>/<service>/ --exclude "*" --include "*<service>*.gz" && 
 find /var/log/<service>/ -iname "*<service>.log.*" -mtime +7 -delete
 endscript

 lastaction
 find /var/log/<service>/ -iname "*<service>.*.log" -delete
 endscript

Yes, I could just make a cron job. My reasoning for using logrotate is that I keep all of the log rotation config management in a single Ansible role and in the future, I'll be making the case to disable log4j rotation and use logrotate instead which will be a simple change.

The Log4j Rolling File Appenders and the logrotate are not compatible. The log4j rotation will interfere with logrotate.

First you need to find the log4j configuration file for the application and replace the Rolling File Appenders with simple File Appenders. This will stop the log4j from rotating the files. Then you can use the logrotate to rotate the log files as you wanted.

If you can't modify the application's log4j configuration you can create your own one and specify this in the classpath when the application is stared.

-Dlog4j.configurationFile=path/to/log4j2.xml

Ref: https://logging.apache.org/log4j/2.0/faq.html#config_location