Securing Log Server in RHEL

Few weeks back I took up the task to replace my syslog server in RHEL5.2 with the new rsyslog package. Redhat packaged rsyslog from RHEL 5 starting with update 2. So I thought of testing it out with stunnel supporting me encryption over the communication line.

The setup goes something like this:-

I have two RHEL5.2 machines one is station1 and the other is server1. The station1 machine sends the log for local6 facility of any type of priority to server1. But the log send over to server1 is going to be encrypted via stunnel package. Let's see how:-

Setup at server1

This is going to be our central log server for local6 log facility. First of all we will install the rsyslog package which though comes with RHEL5.2 but is not the default:-

#yum install rsyslog
#service syslog stop
#yum remove sysklogd
#service rsyslog start
#chkconfig rsyslog on


Next we configure rsyslog such that it listens for connections on tcp/61514

#vi /etc/sysconfig/rsyslog


Edit it such that at line 6 it shows:-

SYSLOGD_OPTIONS="-m 0 -t 61514"


Now we need to add this port 61514/tcp to our semanage ports. This will be done via the following command:-

#semanage port -a -t syslogd_port_t -p tcp 61514


Later we can see if the above command have succesfully worked or not by issuing the following command:-

#semanage port -l | grep syslogd_port_t


The output of the above command will be something like this on a default installation of RHEL5.2

syslogd_port_t tcp 61514
syslogd_port_t udp 514


This tells that port 514/udp and port 61514/tcp are SELinux managed for the type syslogd_port_t. Okay that's what we wanted. For securing the log server we want it to run on a tcp port and that's why we did all this starting from editing /etc/sysconfig/rsyslog to semanage. Note that all our setups have SELinux in enforcing mode so it's necessary that we take proper care of SELinux.

Next we restarted the rsyslog service.

#service rsyslog restart


Now we need to configure stunnel on server1 so that it accepts connections from the client on some fix port and forward them to port 61514/tcp running on server1. We will ensure via the iptables that the port 61514/tcp is not directly exposed to the network as well as port 514/udp.

#iptables -A MYCHAIN -p tcp --dport 60514 -j ACCEPT
#service iptables save


This rule opens up port 60514/tcp on server1. This will be the port where stunnel running on server1 will listen for client connections and later forward them to locally running rsyslog service at 61514/tcp.

The package for stunnel was installed default in a base installation of RHEL5.2 so that was not a big deal but if it's not there in your setup ensure that you have stunnel installed.

After the installation is done we need to configure stunnel. The configuration directory for stunnel is empty but it's package provide one sample conf file which can be used. To use the provided conf sample just follow the below commands:-

#cd /etc/stunnel
#cp /usr/share/doc/stunnel-4.15/stunnel.conf-sample stunnel.conf


Next we edited the stunnel.conf file according to our requirements and when we completely edited it that's how it looked:-


; Certificate/key is needed in server mode and optional in client mode
cert = /etc/stunnel/stunnel.pem
key = /etc/stunnel/stunnel.key

; Some security enhancements for UNIX systems - comment them out on Win32
chroot = /var/run/stunnel/
setuid = nobody
setgid = nobody
; PID is created inside chroot jail
pid = /stunnel.pid

; Some performance tunings
socket = l:TCP_NODELAY=1
socket = r:TCP_NODELAY=1

; Authentication stuff
verify = 2
; It's often easier to use CAfile
CAfile = /etc/stunnel/cacert.pem

; Service-level configuration
[ssyslog]
accept = 60514
connect = 61514


The section [ssyslog] specify which port stunnel will listen to and then which port it will forward the connection too. The destination port is of the local interface (127.0.0.1) as far as I know, haven't digged much into it so I am not sure. Please feel free to comment on it.

There are number of other variables on top that configures alot of stuff. First is the filepath for the stunnel security certificate, then is the filepath for the stunnel security certificate key, next comes the directory under which stunnel will run (this makes stunnel run in a chroot jail, that's good for security reason but it's only available on windows host), after that the user and group with which the application will run and the pid file name and path for stunnel it actually is /stunnel.pid but that's relative to /var/run/stunnel now, after that we had some performance tuning options which actually came enabled default in the sample conf file so I thought of keeping them up, after that verify=2 is used to verify the other end of the tunnel, the verification is done by checking the security certificate of the other end of the tunnel upto depth level 2 so that checks whether the security certificate of the other end (the client end, in our case station1) is actually signed by the same Certificate Authority (CA) as the one specified by the next option that is CAfile.

Now we need to create the directory in which stunnel will store it's pid file and will also run in chrooted jail provided by that directory. The group/owner permission of that directory are also important (as specified in stunnel.conf):-

#mkdir /var/run/stunnel
#chown nobody:nobody /var/run/stunnel


Now we need to work on the security certificate stuff. Stunnel uses both self-signed or third party signed certificates. We went with the trusted third party signed certificate. For this we already had a private Certificate Authority running in our network which was used to sign/revoke security certificates of clients in the network.

So first of all we created the key to be used for the certificate and then we generated a certificate signing request for the stunnel certificate and later send that to the certificate authority to sign and return back to us. The certificate authority also sent us a copy of there own certificate which was also kept in /etc/stunnel for configuration purposes. The following command helped in the above task:-

#cd /etc/stunnel
#openssl genrsa -out stunnel.key 2048
#openssl req -new -key stunnel.key -out stunnel.csr
#scp stunnel.csr root@certificate.example.com:/etc/pki/CA


At certificate.example.com we issued the following commands:-

#cd /etc/pki/CA
#openssl ca -in stunnel.csr -out stunnel.pem
#scp stunnel.pem cacert.pem root@server1.example.com:/etc/stunnel/
#rm -f stunnel.*


Note that after we have recieved the signed certificate and CA certificate the first thing we did was secure those by strictly changing there file permissions as shown below:-

#chown root:root /etc/stunnel/*
#chmod 600 /etc/stunnel/*


That was sufficient. Well if you are not running a local CA I would suggest you do run it or have a commerical 3rd party trusted authority sign your certificate. For a small setup self sign certificate will do the job so no need for Certificate authority. Also note that the step I mentioned above are completely custom as I want them to be it might be that your setup is different then you have to use different commands and options.

That's all about stunnel on the server side. Now was the time to start the tunnel, so that's done just by running the command stunnel.

#stunnel
#ps aux | grep stunnel


The first command runs the tunnel and the next command is given to make sure if stunnel is running in the background successfully or not. The output should be something like this

nobody 4476 0.0 0.3 5060 984 ? Ss 16:46 0:00 stunnel


If you want to make sure that stunnel runs automatically on every boot up just put these lines in /etc/rc.d/rc.local of your system (at the bottom):-

/usr/sbin/stunnel


That's it the job at server1 is done and now it's time to proceed at the client side.

Setup at station1

First the same steps as performed on server1 installing rsyslog package and removing the stock sysklogd package via the following commands:-

#yum install rsyslog
#service syslog stop
#yum remove sysklogd
#service rsyslog start
#chkconfig rsyslog on


Now we need to configure the stunnel package on the client side too. As mentioned earlier stunnel comes with the default installation of RHEL5.2 but if it's not installed just make sure you have it installed. Stunnel actually is part of official RHEL5.2 distribution. Next as done earlier copy the sample configuration file provided by the stunnel package to the stunnel configuration directory.

#cd /etc/stunnel
#cp /usr/share/doc/stunnel-4.15/stunnel.conf-sample stunnel.conf


Edit the file such that it looks as shown below:-

; Certificate/key is needed in server mode and optional in client mode
cert = /etc/stunnel/stunnel.pem
key = /etc/stunnel/stunnel.key

; Protocol version (all, SSLv2, SSLv3, TLSv1)
sslVersion = SSLv3

; Some security enhancements for UNIX systems - comment them out on Win32
chroot = /var/run/stunnel/
setuid = nobody
setgid = nobody
; PID is created inside chroot jail
pid = /stunnel.pid

; Some performance tunings
socket = l:TCP_NODELAY=1
socket = r:TCP_NODELAY=1

; Authentication stuff
verify = 2
; It's often easier to use CAfile
CAfile = /etc/stunnel/cacert.pem

; Use it for client mode
client = yes

; Service-level configuration
[ssyslog]
accept = 127.0.0.1:61514
connect = 192.168.122.2:60514


The major difference between the stunnel.conf of station1 and server1 is that the stunnel.conf of station1 contains a variable client = yes that differentiates server end and client end in a stunnel.

First we will create the chroot directory in which stunnel will run. As done in the configuring the server section above:-

#mkdir /var/run/stunnel
#chown nobody:nobody /var/run/stunnel


Now it's time to make the security certificate for this end of the tunnel. We will proceed in the same way as we did while setting up the server end. First we will generate a 2048 bit key. One particular thing which I forgot to mention about this key is that it's not a password protected key. If it's compromised that end of the tunnel is compromised. We could have protected the key with a password by specifying option like -des3 to the genrsa command but then we would have to give the password for the key when we ran stunnel that asks alot of overhead when we say our tunnel will automatically start on boot. In that case we have to manually feed in the password for the tunnel to get started.

#cd /etc/stunnel
#openssl genrsa -out stunnel.key 2048
#openssl req -in -key stunnel.key -out stunnel.csr
#scp stunnel.csr root@certificate.example.com:/etc/pki/CA


At certificate.example.com the following commands were issued:-

#cd /etc/pki/CA
#openssl ca -in stunnel.csr -out stunnel.pem
#scp stunnel.pem cacert.pem root@station1.example.com:/etc/stunnel
#rm -f stunnel.*


Now as we did during setting up the server end we secure the configuration file and the certificate files at the client end by modifying the file permissions accordingly:-

#chown root:root /etc/stunnel/*
#chmod 600 /etc/stunnel/*


Now we can start the stunnel at the client end too via the simple command stunnel. If we want to start the tunnel automatically on every boot up it's simple just add the line /usr/sbin/stunnel in /etc/rc.d/rc.local at the end. To verify that stunnel is running properly or not just issue the old command ps aux | grep stunnel and see if there is any process owned by user nobody with the name stunnel.

Now we will configure the rsyslog service at the client so that it re-directs all the logs for local6 facility to 127.0.0.1:61514 where stunnel will read them up and send them to 192.168.122.2:60514. Note here that 192.168.122.2 is actually server1 but instead of specifying the name I preferred IP address as DNS can be un-available in my setup.

The below line I added in /etc/rsyslog.conf:-

local6.* @@127.0.0.1:61514


Save and exit and then restart rsyslog:-

#service rsyslog restart


Now to test the setup we issued the following command at station1 and while that was running we sniff the packet via wireshark available in RHEL5.2 to intercept what was getting transferred between the two tunnel ends:-

logger -i -p local6.info -t deependra "This is a test log message sent over stunnel"


The output was clearly seen at server1 in /var/log/messages as

Jul 28 18:24:30 station1 deependra[3460]: This is a test log message sent over stunnel


While the communication was happening between the two ends of the tunnel I sniffed the packets transferred between the two ends and it was all encrypted from what I saw.

That's how I was able to secure my log server communication with clients. There are much better and inbuilt ways to provide security of log server which comes with rsyslog that you can check out at rsyslog website.

NTP Server

It's been long since I am using NTP server in my installations here. So I thought to document my setup a bit in order to explain myself what's going on and to help others world wide so that they can also have a secure setup for Time server in Linux.

Time server is a important part of a network as everybody might be knowing. It is a must if we want to have a network setup which will later consists of kerberos or DNSSEC. It is also needed in windows environment but the configuration for that need not be done in the default case.

I have my test server running latest updated version of Fedora 9. First of all I make sure that my setup have the ntp package. Actually ntp comes default with Fedora distribution so I had no problem in getting the package.

Next step was to make sure I have the correct configuration file setup. So I took a backup of the original file that is /etc/ntp.conf first.

mv /etc/ntp.conf /etc/ntp.conf.bak

Next I wrote the following in a new /etc/ntp.conf

server 127.127.1.0
fudge 127.127.1.0 stratum 1
crypto pw redhat randfile /dev/urandom
keysdir /etc/ntp
restrict default ignore
restrict 127.0.0.1
restrict 192.168.122.0 mask 255.255.255.0 nomodify noquery
driftfile /var/lib/ntp/drift


I know the above options are not the best of the breed but I will explain. First of all if we used our local hardware clock as the time source and then declared it to be at stratum 1 via the fudge line. That may sound madness to everybody. This was done just for testing purpose. Don't do this in your production servers. Please use reliable time source which can be found at http://support.ntp.org/bin/view/Servers/StratumTwoTimeServers

The next line is the crypto line which tells that my ntpkey files are protected with a password redhat and that the filesource which is used for generating random seed data is /dev/urandom. Note that the password attribute is a important one so this file which /etc/ntp.conf should have a strict permission.


#chmod 640 /etc/ntp.conf
#chown root.ntp /etc/ntp.conf


Now the next line tells the directory where all the ntpkey_* files are stored. In fedora 9 it defaults to /etc/ntp/crypto but I used /etc/ntp which is default in RHEL 5.

Next three lines controls the access to the NTP server. The first of them restricts everybody to use the time server or remotely configure the time server. Next restrict line opens up restrictions for the local interface that is 127.0.0.1 This address can do anything no restrictions apply on it. The last restrict line opens the network 192.168.122.0/24 to use the time server to get time service but it can't modify or query (status query on time server) the time server itself. That means any client in the 192.168.122.0/24 can configure 192.168.122.1 as it's reliable time source but it can't use to connect to that server via ntpq or ntpdc utility.

The last line specify the file name which contains the latest estimate of clock frequency error. This file is owned by ntp user.

In the next step we switch to directory /etc/ntp and generate the host keys and IFF parameters as we are going to use IFF identity scheme in this setup.


#cd /etc/ntp
#ntp-keygen -T -I -p redhat


The above command generates the key files and IFF parameters file. The host key file is protected with a password redhat that we also mentioned in /etc/ntp.conf. The list of files which were generated in my case are listed below


ntpkey_cert_station1.example.com
ntpkey_IFFpar_station1.example.com.3426211635
ntpkey_RSA-MD5cert_station1.example.com.3426211635
ntpkey_host_station1.example.com
ntpkey_iff_station1.example.com
ntpkey_IFFkey_station1.example.com.3426211635
ntpkey_RSAkey_station1.example.com.3426211635


In the above list some are key files and some are symbolic links to them. Next we need to extract the IFFkey so that it can transferred to every NTP clients of this server. We can also protect this key with a password that only we and the NTP client knows.


#ntp-keygen -e -q redhat -p linux > ntpkey_IFFkey_station1.example.com.3426211635
#scp ntpkey_IFFkey_station1.example.com.3426211635 root@server1.example.com:/etc/ntp


The above command generate the IFFkey file but the IFF parameter file itself is protected by a password which we specified in the first ntp-keygen command so with -q we specified that password and with -p we specified the password with which the IFFkey file will be protected (the client needs to know this password). The -e option is used to export the IFFkey.

Now I started the ntpd service and configured it to start automatically at the next boot up. Also I had a custom chain in my iptable based firewall in which I opened the udp/123 port on which ntpd listens.

#service ntpd start
#chkconfig ntpd on
#iptables -A MYCHAIN -p udp --dport 123 -j ACCEPT
#service iptables save

Next was the setup at client side that was pretty easy. First of all I configured as usual the main configuration file /etc/ntp.conf

#chmod 640 /etc/ntp.conf
#chown root.ntp /etc/ntp.conf
#vi /etc/ntp.conf


The client side ntp.conf contained the following:

server station1.example.com iburst autokey
crypto pw linux randfile /dev/urandom
keysdir /etc/ntp


The above lines specify the preferred time server to use be station1.example.com aka 192.168.122.1. The option autokey enables the use of public key cryptography. The next line specify the crypto password with which the client ntpkey_* files will be protected and also specify the random seed source to be used. Next line specify where to find the key data.

Next we generated the client side parameters by the following commands

#cd /etc/ntp/
#ntp-keygen -H -p linux
#ln -s ntpkey_IFFkey_station1.example.com.3426211635 ntpkey_iff_station1.example.com
#ln -s ntpkey_host_server1.example.com ntpkey_iff_server1.example.com


The above generates the host parameters on the client side protected by the password linux and next create some symlinks which later configure the IFF keys at the client side. Note here that the file ntpkey_IFFkey_station1.example.com.3426211635 was sent by the time server which was protected by the password linux.

The list of file with the prefix ntpkey_ in there name at the client side /etc/ntp were finally:-


ntpkey_cert_server1.example.com
ntpkey_host_server1.example.com
ntpkey_IFFkey_station1.example.com.3426211635
ntpkey_iff_server1.example.com
ntpkey_iff_station1.example.com
ntpkey_RSAkey_server1.example.com.3426211933
ntpkey_RSA-MD5cert_server1.example.com.3426211933


Now we started the time service at the client and configured it to start automatically at boot and also open the udp/123 port.


#ntpdate -b station1.example.com
#service ntpd start
#chkconfig ntpd on
#iptables -A MYCHAIN -p udp --dport 123 -j ACCEPT
#service iptables save


The first command in the above code was issued to first synchronize the clock of the client with that of the server then start the time service to later keep that new time in synchronization with the server. It took approx. 5 minutes to get synchronized and after that when issued the following command the output was:


#ntpq -cas

ind assID status conf reach auth condition last_event cnt
===========================================================
1 28241 f624 yes yes ok sys.peer reachable 2

#ntpq -c"rv 0 cert"
assID=0 status=0664 leap_none, sync_ntp, 6 events, event_peer/strat_chg,
cert="server1.example.com station1.example.com 0x6",
expire=200907280654,
cert="station1.example.com station1.example.com 0x7",
expire=200907280527, cert="server1.example.com server1.example.com 0x2",
expire=200907280532

#ntpq -c"rv 28241 flags"
assID=28241 status=f624 reach, conf, auth, sel_sys.peer, 2 events, event_reach,
flags=0x83f21


The last command issued returned the flags as 0x83f21 that signifies that the communication with the time server was successful and that IFF identity scheme with cryptography enabled was used.

Client side utilities to check the time configuration are ntpq,ntptrace,ntpdate,nptdc,ntpstat etc.