SPY HILL Research
Spy-Hill.net
Poughkeepsie, New York [DIR] [UP]

Log Rotation for BOINC projects

As your project runs the log files will grow, which may fill up your disk. You will therefore want to rotate your logs, so that you only keep the most recent versions. If your system uses the "logrotate" package then you can easily make use of it to rotate your BOINC log files, but there are also other options.
Last modified: 28 March 2010
Topics
using logrotate
newlog script
web server logs
As your project runs the log files will grow. To keep your disk from filling up and disabling your server it is a good idea to "rotate" your log files. At some interval (every day, week or month) you move your current log file out of place -- usually changing the name to include a sequence number -- and then create a new empty file. You generally keep only a small number of the back logs. This allows you to go back and look through the old logs for a recent time interval, but it keeps the logs from growing without bound.

There are several ways to rotate your log files, but they all have three primary points in common:

  1. Periodic task: You need some way to run a task periodically to rotate the logs, or at least to check whether they are ready to be rotated.
  2. Configuration: You need a configuration file which specifies which files should be rotated and how often, how many back versions to keep, and perhaps other information about the log files (permissions and ownerships)
  3. Rotation program: You need a program or script to do the actual rotation of the logs.
The standard way in Unix to run a periodic task is to use the cron program, which runs as a daemon and runs all sorts of periodic tasks for your machine. For a BOINC project you can also set up periodic tasks in the config.xml file. (The script which runs these tasks is in turn run by cron, but it may be easier to have all your BOINC project configuration in one file or directory.)

Your log rotation policy is generally controled by how often you rotate the logs and how many back copies you keep. For example, if you rotate the logs weekly and keep 9 back copies then you can always go back and look at the logs for about the past two months. Depending on how tight disk space is or how often you need to refer to the back logs you will also choose whether or not to compress the old log files.

A now common tool for rotating log files is the logrotate package. It is likely that this is already installed on your server system. If not, it is relatively easy to install [really? links!?] Alternatives to using logrotate are to use my newlog script described below, to use Ian Patterson's bash script, alas confusingly also called logrotate (http://iain.cx/src/logrotate/) or to write your own script which serves a similar purpose.

Documentation on rotating logs may also be found on the BOINC site:

BOINC logs are written to the directory PROJECT/log_hostname where the "hostname" is the name of your server. If, for performance or storage reasons you want to have them elsewhere (such as /var/log/boinc), then you can put soft links in the log_hostname directory pointing to the "real" log files. In that case you should rotate the logs where they actually reside, not the directory containing the soft links.

logrotate

The logrotate package is now commonly used on many Unix systems to perform periodic rotation of system logs. If you have this on your system, or if you want to install it, this can be an easy way to insure that your logs are properly rotated.

The recommended way to use logrotate is to invoke the program as a periodic task in the BOINC config.xml file. Here is how:

  1. Periodic task: In the config.xml file for your BOINC project include this daily task to invoke the logrotate program:
    <task> <host>alvarez</host> <cmd> /usr/sbin/logrotate -s ../pid_alvarez/logrotate.state ../logrotate.conf <output>cron.log</output> <period> 24 hr </period> </task>
    This example is run on the machine named "alvarez" -- you should change this as appropraite for your own project. The task is run from the PROJECT/bin directory, and so it finds the configuration file ../logrotate.conf in the project directory above it. You could schedule this task more frequently, but the logwatch program will only rotate the logs when it's time to do so.

    You need the "-s" flag to specify an alternate place to store the state file, because you are running logrotate as a non-root user and won't be able to write state information to the system default state file. Be sure to change the name of the directory in which this state file is kept.

  2. Configuration: Your log rotation policies are put in a configuration file called logratate.conf in the project directory. Here is an example configuration file.

    You will need to change the project directory from "/usr02/pirates" to your own project directory, and the name of your machine from "alvarez" to your own server name. You should also change the user name "boincadm" to the name of the account on your server which manages your project (and hence should own the log files). The log files are put in the "apache" group and are group writeable so that the web server can write to them, though this is not necessary for most of them. For security the files are not readable by anybody who is not the owner or in the "apache" group.

    As a general rule, the first directives in the configuration file are global and apply to all log files, but those associated with a particular set of log files override the global settings. (There are some settings which can only be associated with a log file and cannot be global.) The various settings are explained in the Unix man page for logrotate.

    As your project runs you will see which logs need to be rotated more or less frequently. This will depend in some part upon the debug level you specify for each daemon (with the "-d" flag). Higher debug levels will produce more output and make your log files grow more quickly. Simply edit the logrotate.conf file to change your policy as you deem appropriate. The example file should make it clear how to do this.

  3. Rotation program: This method invokes /usr/sbin/logrotate directly. You should specify the full path to the program, which may be different on your server.
It appears that if you select compression for your log files in your logrotate configuration then it will compress files as they are rotated out and will rotate compressed versions of the files, but will not rotate previous uncompressed logs. [Check this more.] The newlog script described below will rotate compressed or uncompressed files, but it doesn't do any compression itself.

 

If you have complete control of your server (ie. "root" access) and logrotate is already being used to rotate your server logs, then an alternative to running logrotate as a BOINC task is to create a file called boinc.conf in the directory /etc/logrotate.d (or wherever your system gets logrotate configurations). Then the BOINC logs listed in that file will be rotated automatically along with your other system files. If you do this, though, you must remove the pre/post-rotation commands to stop and restart your project. The start command would be run as root, causing the lock files for the BOINC daemons to be owned by root, and thus you will not be able to remove or alter them as a non-root user (such as "boincadm"). This is the reason this method is less useful than the other [but it's not clear that restarting the project is important].

Log rotation with newlog

A long while ago I wrote a general log rotation script called newlog, and so I've been using that for my BOINC project. Here's how I have it set up.
  1. Periodic task: In my config.xml file I have a daily task to rotate logs:
    <task> <host>alvarez</host> <cmd> rotate_logs </cmd> <output>cron.log</output> <period> 24 hr </period> </task>

  2. Configuration: This daily BOINC task runs a script in PROJECT/bin called rotate_logs, which simply contains:
    #!/bin/sh PROJECT=/usr02/pirates/ cd $PROJECT/log_`hostname` $PROJECT/bin/newlog feeder.log 9 $PROJECT/bin/newlog cgi.log 9 $PROJECT/bin/newlog transitioner.log 9 $PROJECT/bin/newlog file_deleter.log 9 $PROJECT/bin/newlog sample_dummy_assimilator.log 9 $PROJECT/bin/newlog sample_trivial_validator.log 9 $PROJECT/bin/newlog cron.log 9 $PROJECT/bin/newlog update_stats.log 9 $PROJECT/bin/newlog daily.log 9
    This names the files to be rotated, and says that we should keep 9 back copies of each file. You can obviously add other files to the list to be rotated, or change the number of back copies to keep.

    You could also have separate tasks and scripts to rotate some logs weekly or monthly instead of daily.

  3. Rotation program: The newlog script takes the name of the log file to be rotated and an (optional) number of previous versions to keep. It preserves file permissions and ownerships, or you can force file permissions with an additional argument. It is too long to list here, but you can get it at http://www.spy-hill.net/pub/myers/src/adm/newlog
This should all work on any flavor of Unix -- I wrote the newlog script back before logrotate was common, for use on NextStep, HPUX, Solaris and Linux, and I made it general enough to run correctly on all of these.

Web server logs

If logrotate is installed on your system by default it is likely being used to rotate your web server logs. If the web server is only being used for your BOINC project and your log files are in the "standard" place (usually /var/log/http) then you don't need to do anything, except perhaps review and edit the policy in the configuration file.

If your BOINC project is running as a "virtual host" in addition to the primary web server (as is the case for Pirates@Home) then you can easily rotate the project web logs by adding the log files to the list of files to rotate in the existing logrotate configuration file. Simply edit the file /etc/logrotate.d/httpd and add the access and error logs to the list of files, or use a wildcard (limited to those files, as shown below).

Here is the file /etc/logrotate.d/httpd used on the Pirates@Home server:

/var/log/httpd/*log /usr02/pirates/log_*/http_*.log { weekly rotate 13 missingok notifempty sharedscripts postrotate /bin/kill -USR1 `cat /var/run/httpd.pid 2>/dev/null` 2> /dev/null || true endscript }
The post-rotation 'kill' command is required to send a signal to the running httpd daemon to tell it to close and re-open its log files. Otherwise it will continue to write to the old log files even after they have been given a new name.

If your web server logs are not rotated by logrotate then you might want to find out how they are being rotated and simply add your project web logs to the list of logs to be rotated by that method.
  Copyright © 2010 by Spy Hill Research http://www.Spy-Hill.net/myers/help/boinc/log_rotation.html (served by Islay.spy-hill.com) Last modified: 28 March 2010