Securing Log Server in RHEL

Few weeks back I took up the task to replace my syslog server in RHEL5.2 with the new rsyslog package. Redhat packaged rsyslog from RHEL 5 starting with update 2. So I thought of testing it out with stunnel supporting me encryption over the communication line.

The setup goes something like this:-

I have two RHEL5.2 machines one is station1 and the other is server1. The station1 machine sends the log for local6 facility of any type of priority to server1. But the log send over to server1 is going to be encrypted via stunnel package. Let's see how:-

Setup at server1

This is going to be our central log server for local6 log facility. First of all we will install the rsyslog package which though comes with RHEL5.2 but is not the default:-

#yum install rsyslog
#service syslog stop
#yum remove sysklogd
#service rsyslog start
#chkconfig rsyslog on


Next we configure rsyslog such that it listens for connections on tcp/61514

#vi /etc/sysconfig/rsyslog


Edit it such that at line 6 it shows:-

SYSLOGD_OPTIONS="-m 0 -t 61514"


Now we need to add this port 61514/tcp to our semanage ports. This will be done via the following command:-

#semanage port -a -t syslogd_port_t -p tcp 61514


Later we can see if the above command have succesfully worked or not by issuing the following command:-

#semanage port -l | grep syslogd_port_t


The output of the above command will be something like this on a default installation of RHEL5.2

syslogd_port_t tcp 61514
syslogd_port_t udp 514


This tells that port 514/udp and port 61514/tcp are SELinux managed for the type syslogd_port_t. Okay that's what we wanted. For securing the log server we want it to run on a tcp port and that's why we did all this starting from editing /etc/sysconfig/rsyslog to semanage. Note that all our setups have SELinux in enforcing mode so it's necessary that we take proper care of SELinux.

Next we restarted the rsyslog service.

#service rsyslog restart


Now we need to configure stunnel on server1 so that it accepts connections from the client on some fix port and forward them to port 61514/tcp running on server1. We will ensure via the iptables that the port 61514/tcp is not directly exposed to the network as well as port 514/udp.

#iptables -A MYCHAIN -p tcp --dport 60514 -j ACCEPT
#service iptables save


This rule opens up port 60514/tcp on server1. This will be the port where stunnel running on server1 will listen for client connections and later forward them to locally running rsyslog service at 61514/tcp.

The package for stunnel was installed default in a base installation of RHEL5.2 so that was not a big deal but if it's not there in your setup ensure that you have stunnel installed.

After the installation is done we need to configure stunnel. The configuration directory for stunnel is empty but it's package provide one sample conf file which can be used. To use the provided conf sample just follow the below commands:-

#cd /etc/stunnel
#cp /usr/share/doc/stunnel-4.15/stunnel.conf-sample stunnel.conf


Next we edited the stunnel.conf file according to our requirements and when we completely edited it that's how it looked:-


; Certificate/key is needed in server mode and optional in client mode
cert = /etc/stunnel/stunnel.pem
key = /etc/stunnel/stunnel.key

; Some security enhancements for UNIX systems - comment them out on Win32
chroot = /var/run/stunnel/
setuid = nobody
setgid = nobody
; PID is created inside chroot jail
pid = /stunnel.pid

; Some performance tunings
socket = l:TCP_NODELAY=1
socket = r:TCP_NODELAY=1

; Authentication stuff
verify = 2
; It's often easier to use CAfile
CAfile = /etc/stunnel/cacert.pem

; Service-level configuration
[ssyslog]
accept = 60514
connect = 61514


The section [ssyslog] specify which port stunnel will listen to and then which port it will forward the connection too. The destination port is of the local interface (127.0.0.1) as far as I know, haven't digged much into it so I am not sure. Please feel free to comment on it.

There are number of other variables on top that configures alot of stuff. First is the filepath for the stunnel security certificate, then is the filepath for the stunnel security certificate key, next comes the directory under which stunnel will run (this makes stunnel run in a chroot jail, that's good for security reason but it's only available on windows host), after that the user and group with which the application will run and the pid file name and path for stunnel it actually is /stunnel.pid but that's relative to /var/run/stunnel now, after that we had some performance tuning options which actually came enabled default in the sample conf file so I thought of keeping them up, after that verify=2 is used to verify the other end of the tunnel, the verification is done by checking the security certificate of the other end of the tunnel upto depth level 2 so that checks whether the security certificate of the other end (the client end, in our case station1) is actually signed by the same Certificate Authority (CA) as the one specified by the next option that is CAfile.

Now we need to create the directory in which stunnel will store it's pid file and will also run in chrooted jail provided by that directory. The group/owner permission of that directory are also important (as specified in stunnel.conf):-

#mkdir /var/run/stunnel
#chown nobody:nobody /var/run/stunnel


Now we need to work on the security certificate stuff. Stunnel uses both self-signed or third party signed certificates. We went with the trusted third party signed certificate. For this we already had a private Certificate Authority running in our network which was used to sign/revoke security certificates of clients in the network.

So first of all we created the key to be used for the certificate and then we generated a certificate signing request for the stunnel certificate and later send that to the certificate authority to sign and return back to us. The certificate authority also sent us a copy of there own certificate which was also kept in /etc/stunnel for configuration purposes. The following command helped in the above task:-

#cd /etc/stunnel
#openssl genrsa -out stunnel.key 2048
#openssl req -new -key stunnel.key -out stunnel.csr
#scp stunnel.csr root@certificate.example.com:/etc/pki/CA


At certificate.example.com we issued the following commands:-

#cd /etc/pki/CA
#openssl ca -in stunnel.csr -out stunnel.pem
#scp stunnel.pem cacert.pem root@server1.example.com:/etc/stunnel/
#rm -f stunnel.*


Note that after we have recieved the signed certificate and CA certificate the first thing we did was secure those by strictly changing there file permissions as shown below:-

#chown root:root /etc/stunnel/*
#chmod 600 /etc/stunnel/*


That was sufficient. Well if you are not running a local CA I would suggest you do run it or have a commerical 3rd party trusted authority sign your certificate. For a small setup self sign certificate will do the job so no need for Certificate authority. Also note that the step I mentioned above are completely custom as I want them to be it might be that your setup is different then you have to use different commands and options.

That's all about stunnel on the server side. Now was the time to start the tunnel, so that's done just by running the command stunnel.

#stunnel
#ps aux | grep stunnel


The first command runs the tunnel and the next command is given to make sure if stunnel is running in the background successfully or not. The output should be something like this

nobody 4476 0.0 0.3 5060 984 ? Ss 16:46 0:00 stunnel


If you want to make sure that stunnel runs automatically on every boot up just put these lines in /etc/rc.d/rc.local of your system (at the bottom):-

/usr/sbin/stunnel


That's it the job at server1 is done and now it's time to proceed at the client side.

Setup at station1

First the same steps as performed on server1 installing rsyslog package and removing the stock sysklogd package via the following commands:-

#yum install rsyslog
#service syslog stop
#yum remove sysklogd
#service rsyslog start
#chkconfig rsyslog on


Now we need to configure the stunnel package on the client side too. As mentioned earlier stunnel comes with the default installation of RHEL5.2 but if it's not installed just make sure you have it installed. Stunnel actually is part of official RHEL5.2 distribution. Next as done earlier copy the sample configuration file provided by the stunnel package to the stunnel configuration directory.

#cd /etc/stunnel
#cp /usr/share/doc/stunnel-4.15/stunnel.conf-sample stunnel.conf


Edit the file such that it looks as shown below:-

; Certificate/key is needed in server mode and optional in client mode
cert = /etc/stunnel/stunnel.pem
key = /etc/stunnel/stunnel.key

; Protocol version (all, SSLv2, SSLv3, TLSv1)
sslVersion = SSLv3

; Some security enhancements for UNIX systems - comment them out on Win32
chroot = /var/run/stunnel/
setuid = nobody
setgid = nobody
; PID is created inside chroot jail
pid = /stunnel.pid

; Some performance tunings
socket = l:TCP_NODELAY=1
socket = r:TCP_NODELAY=1

; Authentication stuff
verify = 2
; It's often easier to use CAfile
CAfile = /etc/stunnel/cacert.pem

; Use it for client mode
client = yes

; Service-level configuration
[ssyslog]
accept = 127.0.0.1:61514
connect = 192.168.122.2:60514


The major difference between the stunnel.conf of station1 and server1 is that the stunnel.conf of station1 contains a variable client = yes that differentiates server end and client end in a stunnel.

First we will create the chroot directory in which stunnel will run. As done in the configuring the server section above:-

#mkdir /var/run/stunnel
#chown nobody:nobody /var/run/stunnel


Now it's time to make the security certificate for this end of the tunnel. We will proceed in the same way as we did while setting up the server end. First we will generate a 2048 bit key. One particular thing which I forgot to mention about this key is that it's not a password protected key. If it's compromised that end of the tunnel is compromised. We could have protected the key with a password by specifying option like -des3 to the genrsa command but then we would have to give the password for the key when we ran stunnel that asks alot of overhead when we say our tunnel will automatically start on boot. In that case we have to manually feed in the password for the tunnel to get started.

#cd /etc/stunnel
#openssl genrsa -out stunnel.key 2048
#openssl req -in -key stunnel.key -out stunnel.csr
#scp stunnel.csr root@certificate.example.com:/etc/pki/CA


At certificate.example.com the following commands were issued:-

#cd /etc/pki/CA
#openssl ca -in stunnel.csr -out stunnel.pem
#scp stunnel.pem cacert.pem root@station1.example.com:/etc/stunnel
#rm -f stunnel.*


Now as we did during setting up the server end we secure the configuration file and the certificate files at the client end by modifying the file permissions accordingly:-

#chown root:root /etc/stunnel/*
#chmod 600 /etc/stunnel/*


Now we can start the stunnel at the client end too via the simple command stunnel. If we want to start the tunnel automatically on every boot up it's simple just add the line /usr/sbin/stunnel in /etc/rc.d/rc.local at the end. To verify that stunnel is running properly or not just issue the old command ps aux | grep stunnel and see if there is any process owned by user nobody with the name stunnel.

Now we will configure the rsyslog service at the client so that it re-directs all the logs for local6 facility to 127.0.0.1:61514 where stunnel will read them up and send them to 192.168.122.2:60514. Note here that 192.168.122.2 is actually server1 but instead of specifying the name I preferred IP address as DNS can be un-available in my setup.

The below line I added in /etc/rsyslog.conf:-

local6.* @@127.0.0.1:61514


Save and exit and then restart rsyslog:-

#service rsyslog restart


Now to test the setup we issued the following command at station1 and while that was running we sniff the packet via wireshark available in RHEL5.2 to intercept what was getting transferred between the two tunnel ends:-

logger -i -p local6.info -t deependra "This is a test log message sent over stunnel"


The output was clearly seen at server1 in /var/log/messages as

Jul 28 18:24:30 station1 deependra[3460]: This is a test log message sent over stunnel


While the communication was happening between the two ends of the tunnel I sniffed the packets transferred between the two ends and it was all encrypted from what I saw.

That's how I was able to secure my log server communication with clients. There are much better and inbuilt ways to provide security of log server which comes with rsyslog that you can check out at rsyslog website.