arlogd - a remote logger
arlogd-server [-h] [-s] [-t] [-c config] [stop|start] arlogd-client [-h] [-s] [-t] [-c config] [stop|start]
Arlogd provides remote logging ; a daemon on the client host transfers data to a daemon on the server host (loghost).
Programs on a client host use remote logging by writing to a named pipe (fifo).
On a client, a daemon periodicly searches directory $fifos
(default /var/log/fifos) for readable fifo's.
Note : a fifo is created with mkfifo /var/log/fifos/my-fifo
Anything written into such a fifo, appears immediately in a corresponding file on the loghost :
client: /var/log/fifos/foo/bar loghost: /var/log/arlog/foo/bar
Start the (loghost or client) daemon with :
The exit status is 0 if a daemon is started (or was already running) ;
1 otherwise ; likewise for stop.
Without an argument, arlogd-server and arlogd-client
show the daemon's status.
arlogd-server arlogd-client
The exit status is 0 if the daemon is running ; 1 otherwise.
The default location for a config-file is
arlogd.conf or /etc/arlogd/conf.
If/when $loghost is unavailable, the client will :
save log-entries in file $save_file (default /var/log/arlogd-save.log) ;
periodicly try to reconnect to $loghost ;
on reconnect, send $save_file to $loghost.
On the server, you can send commands to the daemon ; see ARLOGD-SERVER below.
On the client, the daemon periodicly writes a state-file
$run_dir/client.state containing a list of watched fifo's ;
see ARLOGD-CLIENT below.
See below :
To use remote logging for some logfile, just replace the logfile by a symlink to a fifo :
# remote logging for /var/log/httpd/vhost/access.log
% mkdir -p /var/log/fifos/httpd/vhost/ % mkfifo /var/log/fifos/httpd/vhost/access.log % cd /var/log/apache % mv access.log TEMP % ln -s /var/log/fifos/httpd/vhost/access.log . % cat TEMP > access.log
Things to check or do :
Restart utilities that keep their logfile open. Some (like apache) do, others (like rsync) don't.
fuser TEMP may tell you which processes are still using TEMP.
Remove TEMP.
Don't forget to turn off log processing (like log-rotation
by logrotate(8) or other processes) for the logfile you just
made remote.
Show help and default config ; then exit.
Be silent ; use logs instead of stdout ; used in init.d/ scripts.
Just check the config.
Use config file.
A configuration file must be specified with -c config-file or be present as
./arlogd.conf /etc/arlogd/conf
In the config-file, lines starting with # are skipped ; as are empty
lines ; lines with leading and/or trailing white space are stripped.
Each line contains a name/value pair, separated by white space.
The list below shows the default value for each name. $name indicates the configured value of entry name.
loghost localhost
Clients send their data to host $loghost.
Specify a host.domain name, or an IP address ; eg
loghost loghost.my.org
allow_hosts localhost
The hosts that are allowed to connect to the $loghost.
Specify a white-space-separated list of hosts and/or IPs ; eg
allow_hosts localhost webfarm.my.org monitor.my.org
autoflush 1
By default, the daemon on the loghost writes the logfiles with autoflush on : the daemon flushes files after every write.
Autoflush is expensive, but it will keep the contents of the remote logfiles as up-to-date as possible.
It is probably wise to turn off autoflush later, when things are running smoothly.
To turn autoflush off, configure :
autoflush 0
rename_list /etc/arlogd/renames
The specified file should contain a list of (src,dst)-pairs like :
mv file /path/to/dest
Path $file is used relatative to $log_dir ;
that is, $log_dir/$file should exist.
The $rename_list is used by command RENAME (see below). It attempts to
rename $log_dir/$src $dst
for each (src,dst)-pair.
Renaming can be useful if you want to integrate remote logfile processing (rotation, analysis etc) into your normal process. Use renaming to move your remote logfiles into your normal log-tree, just before normal processing begins.
If the list is long, you should probably generated it ; the list is re-read every time RENAME runs.
log_names *.log
On the loghost, the daemon can manipulate logfiles on demand. Pattern $log_names is the default file name pattern for the daemon.
In pattern, * means string not containing / ;
it is always used to match as a suffix.
port 2207
The loghost data port ; $port+1 is used on $loghost for commands .
logfile /var/log/arlogd.log
The server- and client-daemons log into file $logfile.
run_dir /var/run/arlogd
The server- and client-daemons keep pid-files etc in directory $run_dir.
log_dir /var/log/arlog
The server on $loghost writes the logs in directory $log_dir.
fifos /var/log/fifos
A client-daemon watches named pipes in directory $fifos.
save_file /var/log/arlogd-save.log
If/when the loghost is unavailable, a client saves log-entries in $save_file.
reconnect 300
If/when the loghost is unavailable, a client will try to reconnect to the loghost every $reconnect seconds.
On reconnect, the client sends the contents of $save_file to $loghost.
upd_fifos 60
A client searches the fifos directory, looking for new (or disappeared) fifo's, every $upd_fifos seconds.
Program arlogd-server opens two service ports (default 2207 and 2208), and forks a splitter, which does the actual work.
On port 2207 the daemon accepts data connections from hosts in $allow_hosts. The daemon expects lines like
file-path log-line
The log-line is appended to file $log_dir/file-path.
If the log-line starts with / or contains .. as a path-component,
the log-line is appended to file $log_dir/BAD.log.
On port 2208 the daemon accepts command connections from localhost.
To issue a command, on $loghost (for now) use netcat :
echo command | nc -v localhost 2208
These commands let you get/set the value of autoflush in the running daemon.
Command AUTOFLUSH? gets the value ; it prints :
COMMAND AUTOFLUSH? AUTOFLUSH current_value_autoflush
Commands AUTOFLUSH and NO_AUTOFLUSH set the value of
autoflush, and print :
COMMAND [NO_]AUTOFLUSH AUTOFLUSH old_value -> new_value
The daemon closes all files in $log_dir.
Command CLOSE prints something like :
COMMAND CLOSE FILES 3 FILE: /path/to/file1 FILE: /path/to/file2 FILE: /path/to/file3
If/when the daemon is using (normal) buffered IO ($autoflush is off), and you want to see the actual contents of a remote logfile, then you can use CLOSE to make the daemon close (and thus flush) its open files.
The daemon finds files in $log_dir matching pattern as a suffix.
The default pattern is $log_names (default *.log).
Command FIND prints something like :
COMMAND FIND FILES 3 FILE: /path/to/file1 FILE: /path/to/file2 FILE: /path/to/file3
and possibly some lines like :
ERROR some error
Use FIND to test pattern before you use it in ROTATE.
On PING, the daemon prints :
PONG
PING is for debugging.
The daemon renames files as indicated in file $rename_list
(default /etc/arlogd/renames) ; see config entry
rename_list.
File $rename_list must contain lines like
mv file /path/to/dst
where file is a relative path in $log_dir :
RENAME operates on $log_dir/file.
Lines starting with # are skipped, as are empty lines ;
lines with leading and/or trailing white space are stripped.
Because (a priori) RENAME will move anything anywhere,
we need some constraints :
The owner of the file $rename_list must be the same as the owner
of the (running) daemon executing the RENAME command.
The daemon trusts that its owner knows what he/she is doing.
The $rename_list must not be group- or world-writable. The daemon trusts only its owner.
RENAME will not rename file if file is a directory or a symlink.
RENAME will not rename file to /path/to/dst
if /path/to/dst is a directory or a symlink.
RENAME will not create in-between directories for
/path/to/dst : /path/to/ must exist.
Please note :
Before each rename, /path/to/dst is removed ;
so, file /path/to/dst is always lost.
If file is missing, RENAME attempts to create an empty
file /path/to/dst.
A missing file is assumed to be empty.
If/when you run RENAME daily, and nothing is logged in file
during the previous day, then file will not exist.
If you want to change the location of the $rename_list, you have to reconfigure and restart the daemon ; this is intentional.
Command RENAME prints something like :
COMMAND RENAME FILE: $rename_list FILES 3 DONE: /path/to/src1 /path/to/dst1 DONE: /path/to/src2 /path/to/dst2 DONE: /path/to/src3 /path/to/dst3
and possibly some lines like :
ERROR can't open list (why) ERROR can't rename $file (why) ERROR bad line [$line]
or, if file is missing :
WARN: can't find $file DONE: EMPTY /path/to/dst ERROR can't create empty /path/to/dst (why)
The daemon rotates files in $log_dir matching pattern as a suffix.
The default pattern is $log_names (default *.log).
To rotate X means : close X and rename X as X.rot.
arlog/www.my.org/access.log -> arlog/www.my.org/access.log.rot
A separate utility must further rotate, compress, analyse stuff.
Command ROTATE prints something like :
COMMAND ROTATE FILES 3 FILE: /path/to/file1 FILE: /path/to/file2 FILE: /path/to/file3
and possibly some lines like :
ERROR can't rename /path/to/file
The daemon checks the secret, closes all files in $log_dir and stops. The secret is generated by the daemon on start-up.
The daemon writes the secret to $run_dir/server.stp with mode 0600,
so command STOP can only be used by the user who owns the daemon.
Command STOP prints :
STOPPED
or
BAD SECRET
STOP is used by arlogd-server to cleanly stop the running daemon.
A fifo must not have white space in its name, or path-components. Such fifo's are flagged as BAD. They are watched, but anything read from them will be discarded.
client.stateEvery $upd_fifos seconds (default 60)
the daemon writes file $run_dir/client.state.
It shows where the daemon is writing ($loghost or $save_file)
and a list of watched fifo's in directory
$fifos (default /var/log/fifos).
Installation of arlogd requires very little :
Perl, perl core-modules IO, File::Path, Net::hostent.
logger(1) in /bin, /sbin, /usr/bin or /usr/sbin.
On a loghost, you need netcat(1) if you want to use the daemons
command facility.
Installation is simple :
Checkout the repo :
svn co https://svn.science.uu.nl/repos/project.arlogd/pub/trunk arlogd
On a server (loghost) :
make -f install-server.mk
On a client :
make -f install-client.mk
Arlogd is added as a service, but the service is not turned on or started. To turn on the service use
chkconfig arlogd-server on chkconfig arlogd-client on
To configure, edit /etc/arlogd/conf.
On the server (loghost), configure entry
allow_hosts
(default localhost).
allow_hosts localhost webfarm.my.org monitor.my.org
On a client, configure entry
loghost
(default localhost).
loghost loghost.my.org
On a server (loghost) :
/etc/init.d/arlogd-server start
On a client :
/etc/init.d/arlogd-client start
On the loghost and the client watch the log :
tail -F /var/log/arlogd.log
On the client, make a fifo :
mkfifo /var/log/fifos/test
Within a minute, arlogd-client will start watching this fifo ;
see the log.
Write something to the fifo :
echo something > /var/log/fifos/test
On the loghost, something should immediately appear in file
/var/log/arlog/test
You can also try to create a fifo in a subdirectory ; and write to it immediately :
mkdir -p /var/log/fifos/foo/bar mkfifo /var/log/fifos/foo/bar/test echo something else > /var/log/fifos/foo/bar/test
Expect echo to block ; be patient ; echo is waiting for a reader.
As soon as arlogd-client finds fifo foo/bar :
... echo unblocks ;
and on the loghost, something else appears in
/var/log/arlog/foo/bar/test
On the loghost, you can talk to the daemon if you have netcat nc(1).
echo PING | nc localhost 2208
This should reply PONG.
allow_hosts and loghost
Make sure you have configured allow_hosts (on the loghost) and loghost (on the clients).
The defaults (localhost for both) are probably not sane.
$GREP_OPTIONS --devices=skip
By default, grep(1) reads devices, and will hang
when it encounters a fifo ; for instance, when you do
grep -r some-pattern /var/log/
Grep(1) will silently ignore devices if you set environment variable
GREP_OPTIONS :
$GREP_OPTIONS = --devices=skip
When you do want to grep devices, use
grep --devices=read ...
The daemons should run from (multi-user) boot until shutdown.
Make sure that the arlogd service (server or client) is turned on.
chkconfig --list arlogd-server chkconfig --list arlogd-client
If not, turn on the service with (either) :
chkconfig arlogd-server on chkconfig arlogd-client on
On a client, make sure that arlogd-client is started early,
because a utility that logs to a fifo will block until
arlogd-client is started.
Inspect /etc/rc.d/rc2.d/ and assert that S??arlogd-client is listed
before S??service for all services that use remote logging.
Less important, check that (on shutdown) arlogd-client is stopped
late, because you may lose some log-lines when the daemon
(with autoflush off) is stopped too early.
Inspect /etc/rc.d/rc6.d/ and assert that K??arlogd-client is listed
after K??service for all services that use remote logging.
Note: to see the init.d start/stop priorities of arlogd,
look at /etc/init.d/arlogd-server and /etc/init.d/arlogd-client.
The relevant line looks like :
# chkconfig: 2345 $start_prio $stop_prio
Note: configuration management of the daemon-startup-sequence varies wildly between Linux/Unix platforms and installations ; YMMV ...
Every minute, the client deamon has to walk the $fifos-tree,
looking for fifo's. That's cheap if the tree is flat (has only a
few flat sub-directories) ; the flatter, the better.
But if your client isn't very busy, or /var/log/ isn't very big,
you may want to configure
fifos /var/log
and just replace logfiles with fifo's.
Use find(1) to see how long it takes to walk /var/log,
looking for fifo's :
time find /var/log -type p -print
Maybe having fifo's in /var/log/ (or in /var/log/fifos/)
isn't such a good idea.
Think about grep -r ... /var/log/ ; it will hang ;
although grep -D skip will not (it skips devices).
So, if it is a problem, the default for $fifos should
be something ugly like /dev/arlogd.
Or, if it isn't a problem, the default for $fifos
could be /var/log ; that is preferable because it simplifies
usage for Joe average (``just replace a logfile by a fifo'').
Scanning /var/log/ is cheap, unless it is VERY big.
Try : time find /var/log -type p -print
A production note, could tell users with a BIG /var/log/
that it is more efficient to collect their fifo's in some separate directory,
and use symlinks and/or reconfigure their apps to log elsewhere.
Suppose the daemons are running. And suppose you create a named pipe :
mkfifo -m 0755 /var/log/fifos/foo/bar.log
... and some utility writes a log-line to
/var/log/fifos/foo/bar.log.
Then, on the client, the daemon will :
foo/bar.log (within a minute),
foo/bar.log
On the loghost, the daemon will :
The loghost only checks that the prefix doesn't
start with / (slash) and doesn't contain
.. as a path-component.
On the client, the daemon keeps a list of fifo's in $fifos
(default /var/log/fifos), which it refreshes every minute.
The client daemon watches the fifo's all at once. If/when any fifo has new data, the daemon quickly picks it up and sends it to the loghost.
On the server, the daemon consists of two processes : the server and the splitter.
The server handles incoming connections, and pipes all log-lines (and some commands) to the splitter.
The splitter stores log-lines in the appropriate log-files. It maintains a hash of open log-files, indexed by the prefixes it has received.
The splitter keeps files open, until instructed to close them.
The splitter has a back-pipe to the server ; it is used to send command-results back to the server.
Commands are picked up by the server. Some commands are piped to the splitter.
To distinguish between commands and log-lines, commands have a special prefix and a secret.
The secret is generated when the daemon starts up and shared (only) between the server and the splitter.
When a command must start a daemon, it tries to make sure
the daemon will not run into problems immediately.
Note that the launched daemon is entirely detached from the launcher ; the launcher can't know if the daemon failed (quit) on start-up.
Pre-launch checks include :
Make sure that the configured directories ($log_dir on the server, $fifos on the client, $run_dir on both) exist or can (actually) be created.
In test-mode (option -t), any created directories are removed.
A failure is a fatal.
Unless in test-mode, make sure that lockfile
$run_dir/server.lck (or /client.lck)
can be written and (exclusively) locked. A failure is fatal.
Make sure that the $logfile can be appended ; on failure,
a warning is given and a /tmp/ file is used.
Check that configured hosts are resolvable ($allow_hosts on the server, $loghost on the client) ; on failure, (only) a warning is given.
On the client it makes sense to continue, even if the $loghost is unreachable, because the daemon will watch the fifo's ; ulities won't block for the lack of a fifo-reader.
In option processing, (only) warnings are issued if spurious options are found.
It is a fatal error to supply an argument which isn't start or stop,
or to supply more than one argument.
Commands arlogd-server and arlogd-client set an exit-code.
They may write to stdout.
Using -s (for silent) makes them silent.
The exit-code is 0 if
argument start : a daemon is launched, or was already running.
argument stop : the daemon is stopped, or wasn't running.
no argument : there is a running daemon.
A running daemon may write to $logfile and syslog.
Events logged in syslog, are also logged in $logfile.
If the daemon can't append to $logfile, it uses some /tmp/ file.
Start/stop events are logged in syslog, as are config- and other startup-errors.
The default install does this :
make directory /etc/arlogd/ unless it exists ;
create (with touch) an empty file /etc/arlogd/conf ;
create (with touch) an empty file /etc/arlogd/renames ;
copy the daemon (arlogd.server or arlogd.client) to /usr/sbin ;
copy init.d/arlogd-server (or init.d/arlogd-client) to /etc/init.d ;
add arlogd as a service with :
chkconfig --add arlogd
/etc/arlogd/conf /etc/arlogd/renames /etc/init.d/arlogd-client /etc/init.d/arlogd-server /var/lock/subsys/arlogd /var/log/arlog/ /var/log/arlogd-save.log /var/log/arlogd.log /var/log/fifos/ /var/run/arlogd/client.lck /var/run/arlogd/client.pid /var/run/arlogd/client.state /var/run/arlogd/server.lck /var/run/arlogd/server.pid
© 2014
Henk P. Penning,
Faculty of Science,
Utrecht University
arlogd version arlogd-0.1.4 - Thu Sep 18 12:49:01 2014 UTC - dev revision 102