Sunday 28 February 2016

Centralized Log Management using rsyslog CentOS 6/7

​I am creating an centralized log server where it can store all the logs from the clients. In order to do that make sure you have enough space to store logs for all the clients. I would also configure the log rotation to save space on the disk.

Environment - CentOS/Redhat 6.6
rsyslog version5.8.10

rsyslog would be installed by default. incase its not there use an yum to install.
#yum install rsyslog

It could be helpful incase you read man rsyslog.conf documentation. It has mainly 3 parts 

1. Modules - rsyslog follows modular design
2. Global directives - Set global properties for rsync
3. Rules - what to be logged and where

Destination log server would contain all the logs ( audit, sudo, su, history, kernel, ..etc) to be logged from all the clients to the centralized log server.

Let's configure server first,  

Edit, 
#vim /etc/rsyslog.conf 

#Make sure you have syslog reception as TCP/UDP communication. 
$ModLoad imudp
$UDPServerRun 514

$ModLoad imtcp
$InputTCPServerRun 514

# Create a template so that all the logs would write to respective clients with the name of the program being logged in below destination path. You can keep your priority.facilitator to be marked and would be sent to rsyslog daemon which used to log centrally.

$template TmplAuth,"/scratch/remote-sys-logs/%fromhost%/%PROGRAMNAME%.log"
authpriv.*   ?TmplAuth
*.info,mail.none,authpriv.none,cron.none,local6.*  ?TmplAuth

# Since I needed audit.log there by I would create a new rule to make sure that it would reach to the same destination folder.

$template TmplAudit,"/scratch/remote-sys-logs/%fromhost%/audit.log"
local6.*        ?TmplAudit

# logging all the bash terminal commands and storing in the centralized location.

$template TmplCmds,"/scratch/remote-sys-logs/%fromhost%/hist.log"
local0.debug    ?TmplCmds

save and quit the file.

# mkdir /scratch/remote-sys-logs
# service rsyslog restart

Since the logs would be big, I would like to rotate in such way that two files in which last rotated file would be zipped, whereas last but one not to be zipped. The maximum time I would like to keep the logs are for 60 days. It would store with the date format as a extension. These would be processed once the rsyslog restarts.

Edit, 
#vim /etc/logrotate.d/remote-sys-logs
/scratch/remote-sys-logs/*/*.log {
    daily
    dateext
    rotate 2
    compress
    create 644 root root
    notifempty
    missingok
    maxage 60
    sharedscripts
    postrotate
     /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
    endscript
}

Client: 

Edit. 
vim /etc/rsyslog.conf

# the below module doesn't exist by default, make an entry so that it has an ability to convert any standard text file into a syslog message.
$ModLoad imfile

# Enter the module definition as below to copy logs from audit to centralized log server, failing to do this so would not be able to write to central log.


$InputFileName /var/log/audit/audit.log
$InputFileTag audit:
$InputFileStateFile audit.log
$InputFileSeverity info
$InputFileFacility local6
$InputRunFileMonitor

# forward all the logs to the centralized server. 

*.*                     @centrallogserver:514

save and quit
#service rsyslog restart

In order to log all the bash commands to the logger, make an entry in /etc/bashrc global config file.


#export so that all the new bash commands are being logged to the file
export PROMPT_COMMAND='RETRN_VAL=$?;logger -p local0.debug "$(whoami):[$$] $(history 1 | sed "s/^[ ]*[0-9]\+[ ]*//" ) [$RETRN_VAL]#"'

exit or fork the bash shell to log all the history to the centralized log server.

Hope this helps someone who would like to create central log server. Below are the references that could be checked to suit your requirements.

Thanks for studying and re-sharing !

References :
https://en.wikipedia.org/wiki/Syslog - Syslog facility and priorities explained 
man logrotate.conf
man rsyslog.conf