Syslog-ng

As it is stated in the Gentoo Security Guide, syslog-ng provides some of the same features as syslog and metalog with a small difference. It can filter messages based on level and content (like metalog), provide remote logging like syslog, handle logs from syslogd (even streams from Solaris), write to a TTY, execute programs, and it can act as a logging server. Basically it is the best of both loggers combined with advanced configuration.

This tutorial aims to explain the syntax of the syslog-ng configuration file using examples.

Quick Start
First, you have to emerge syslog-ng and add it to the default runlevel. If you want to start syslog-ng just now, execute the init script. Also, it is recommended to unmerge any system logger you have previously installed as loggers often use.

Also, you may want logrotate to rotate your logs

For a quick start, here there is a classic configuration file slightly modified from the one in the Gentoo Security Guide. You will see your defined log files in. In the following sections we will explain this configuration file in order to understand how syslog-ng works.


 * Updated for Syslog-ng version 3 formatting --Oldr4ver 16:11, 28 February 2010 (GMT)
 * Added Shorewall and Webmin logging to the example configuration
 * Edited filtering example; still not working

Options
Description of the suggested options:
 * chain_hostnames(off): Disable the chained hostname format. Default: no
 * stats_freq (43200): The period between two STATS messages in seconds. STATS are log messages sent by syslog-ng, containing statistics about dropped log messages. Default: 600
 * owner(root): The default owner of output files. By default, syslog-ng changes the privileges of accessed files (e.g., /dev/null) to root.root 0600.
 * group(root): The default group of output files.
 * perm(0640): The default permission for output files
 * dir_perm(0740): The default permission for newly created directories. Default: 0700.
 * create_dirs(yes): Enable or disable directory creation for destination files. Default: no.
 * use_fqdn(no): Add Fully Qualified Domain Name instead of short hostname. Default: no.
 * keep_hostname(yes): Enable or disable hostname rewriting. Enable this option to use hostname-related macros. Default: no.
 * use_dns(no): Enable or disable DNS usage. Default: yes.

Full description of the options can be found in the syslog-ng admin guide.

Destinations
In syslog-ng log messages are sent to files. The syntax is very similar to sources:

You will be normally logging to a file, but you could log to a different destination-driver: pipe, unix socket, TCP-UDP ports, terminals or to specific programs. Therefore, this means sending authlog messages to :

If the user is logged in, usertty sends messages to the terminal of the specified user. If you want to send console messages to root's terminal if it is logged in:

Messages can be sent to a pipe with pipe. The following sends xconsole messages to the pipe. This needs some more configuration, so you could look at the sub-section xconsole below.

To send messages on the network, use udp. The following will send your log data out to another server.

xconsole
Before syslog-ng can send messages to the pipe /dev/xconsole, you should create it with mkfifo, giving the appropriate access and owner permissions.

Afterwards, you need a program to read the messages sent to. xconsole is a console that monitors system messages with X. First step is to emerge xconsole. Tell xconsole to read /dev/xconsole after that.

Text window space and geometry can be modified in the following ways:

Creating Filters for Messages
The syntax for the filter statement is:

Functions can be used in the expression, such as the fuction facility which selects messages based on the facility codes. The linux kernel has a few facilities you can use for logging. Each facility has a log-level; where debug is the most verbose, and panic only shows serious errors. You can find the facilities, log levels and priority names in. To filter those messages coming from authorisation, like  May 11 23:42:31 mimosinnet su(pam_unix)[18569]: session opened for user root by (uid=1000) , use the following:

The facility expression can use the boolean operators and, or, and not, so the following filter selects those messages not coming from authorisation, network news or mail:

The funciont level selects messages based on its priority level, so if you want to select informational levels:

Functions and boolean operators can be combined in more complex expressions. The following line filters messages with a priority level from informational to warning not coming from auth, authpriv, mail and news facilities:

Messages can also be selected by matching a regular expression to the text of the log message, excluding the headers with the function. For example:

To match a regular expression to the headers and the message itself, you can use the function.

Log Paths
Syslog-ng connects sources, filters and destinations with log statements. The syntax is:

The following for example sends messages from 'src' source to 'mailinfo' destination filtered by 'f_info' filter.

Tips and Tricks
After understanding the logic behind syslog-ng, many possible and complex configuration are possible. Here there are some examples.

Failover Logging to Remote Host
This setup shows how to send the default unencrypted syslog packets across both tcp and udp protocols, using the standard port (514) and an alternate port. This is sending the same output to the same machine 4 different ways to try and make sure packets make it. Mostly useful if you are debugging a remote server that fails to reboot. The different ports and protocols are to make it past any firewall filters or other network problems. Also useful for port-forwarding and using tunnels. Something like this setup is ideal to tunnel across an ssh connection that the prone-to-failover host initiates through a reverse connection.

And then on the loghost receiving these logs:

Log directly to MySQL
Syslog-ng directly to MySQL

Move log to another file
In order to move some log from to another file:

Configuring as a loghost
Configuring your system to be a loghost is quite simple. Drop the following into your configuration, and create the needed directory. With this simple configuration, log filenames will be based on the FQDN of the remote host, and located in. After creating the remote directory, reload your syslog-ng configuration.

Improve Performance
Syslog-ng's performance can be improved in different ways:

Avoid redundant processing and disk space
A single log message can be sent to different log files several times. For example, in the initial configuration file, we have the following definitions:

The same message from the 'cron' facility will end up in both the cron.log and messages file. To change this behavior we can use the final flag, ending up further processing with the message. Therefore, in this example, if we want messages from the 'cron' facility not ending up in the messages file, we should change the cron's log sentence by:

another way is to exclude the cron facility from f_messages filter:

Postgresql Destination
This section will use two roles: syslog and logwriter. syslog will be the administrator of the database syslog and logwriter will only be able to add records to the logs table.

No longer needed to create table for logs. Syslog-ng will create automatically.

postgres=# CREATE ROLE syslog WITH LOGIN; postgres=# \password syslog   # Using the \password function is secure because postgres=# \password logwriter # the password isn't saved in history. postgres=# CREATE DATABASE syslog OWNER syslog; postgres=# \q # You're done here for the moment

Edit to allow syslog and logwriter to establish a connection to PostgreSQL.

Tell PostgreSQL to reload the configuration files:

Edit so that it knows where and how to write to PostgreSQL. Syslog-ng will utilize the logwriter role.

There is a problem with the PID fields down here, and needs to be fixed, but the idea still works once you have the libdbi/libdbi-drivers working

Finally, restart Syslog-ng.

And check to see if things are being logged.

syslog=> SELECT * FROM ORDER BY datetime DESC LIMIT 10;