In order for a system administrators to recognize or analyze problems on a CentOS 8 or RHEL 8 server, it is important to know and view the events that occurred on the server in a particular period of time from log files found in /var/log the directory in the system.
The Syslog (System Logging Protocol) system on the server can act as a central log monitoring point over a network where all servers, network devices, switches, routers and internal services that create logs, whether linked to the particular internal issue or just informative messages can send their logs.
On a CentOS/RHEL 8 server, Rsyslog daemon is the most important log server that comes preinstalled by default, followed by Systemd Journal Daemon (journald).
Rsyslog is an open-source utility, developed as a client/server architecture service and can achieve both roles independently. It can run as a server and gather all logs transmitted by other devices over the network or it can run as a client by sending all internal system events logged to a remote Syslog server.
Requirements
In order to set up a centralized log server on a CentOS/RHEL 8 server, you need to check an confirm that the /var partition has enough space (a few GB minimum) to store all recorded log files on the system that send by other devices on the network. I recommend you to have a separate drive (LVM or RAID) to mount the /var/log/ directory.
How to Configure Rsyslog Server in CentOS/RHEL 8
1. As I said, Rsyslog service is installed and running automatically in CentOS/RHEL 8 server. In order to verify that the daemon is running in the system, run the following command.
# systemctl status rsyslog.service
If the service is not running by default, run the following command to start rsyslog daemon.
# systemctl start rsyslog.service
2. If the Rsyslog utility is not installed by default on the system that you plan to use as a centralized logging server, run the following dnf command to install the rsyslog package and start the daemon.
# dnf install rsyslog # systemctl start rsyslog.service
3. Once Rsyslog utility installed, you can now configure rsyslog as a centralized logging server by opening the main configuration file /etc/rsyslog.conf, in order to receive log messages for external clients.
# vi /etc/rsyslog.conf
In the /etc/rsyslog.conf configuration file, find and uncomment the following lines to grant UDP transport reception to the Rsyslog server via 514 port. Rsyslog uses the standard UDP protocol for log transmission.
module(load="imudp") # needs to be done just once input(type="imudp" port="514")
4. The UDP protocol doesn’t have the TCP overhead, and it makes data transmission faster than the TCP protocol. On the other hand, the UDP protocol doesn’t guarantee the reliability of transmitted data.
However, if you want to use TCP protocol for log reception you must find and uncomment the following lines in the /etc/rsyslog.conf the configuration file in order to configure Rsyslog daemon to bind and listen to a TCP socket on 514 port.
module(load="imtcp") # needs to be done just once input(type="imtcp" port="514")
5. Now create a new template for receiving remote messages, as this template will guide the local Rsyslog server, where to save the received messages send by Syslog network clients.
$template RemoteLogs,"/var/log/%HOSTNAME%/%PROGRAMNAME%.log" *.* ?RemoteLogs
The $template RemoteLogs directive guides Rsyslog daemon to gather and write all of the transmitted log messages to distinct files, based on the client name and remote client application that created the messages based on the outlined properties added in the template configuration: %HOSTNAME% and %PROGRAMNAME%.
All received log files will be written to the local filesystem to an allocated file named after the client machine’s hostname and kept in /var/log/ directory.
The & ~ redirect rule directs the local Rsyslog server to stop processing the received log message further and remove the messages (not write them to internal log files).
The RemoteLogs is an arbitrary name given to this template directive. You can use whatever name you want that best suitable for your template.
To configure more complex Rsyslog templates, read the Rsyslog configuration file manual by running the man rsyslog.conf command or consult Rsyslog online documentation.
# man rsyslog.conf
6. After making the above configuration changes, you can restart the Rsyslog daemon in order to apply recent changes by running the following command.
# service rsyslog restart
7. Once you restarted the Rsyslog server, it should now act as a centralized log server and record messages from Syslog clients. To confirm the Rsyslog network sockets, run netstat command and use grep utility to filter rsyslog string.
# netstat -tulpn | grep rsyslog
If netstat command not intall on CentOS 8, you can install it using the following command.
# dnf whatprovides netstat # dnf install net-tools
8. If you have SELinux active in CentOS/RHEL 8, run the following command to allow rsyslog traffic depending on network socket type.
# semanage port -a -t syslogd_port_t -p udp 514 # semanage port -a -t syslogd_port_t -p tcp 514
If semanage command not install on CentOS 8, you can install it using the following command.
# dnf whatprovides semanage # dnf install policycoreutils-python-utils
9. If you have a firewall active on the system, run the following command in order to add the needed rules for allowing rsyslog traffic on ports in Firewalld.
# firewall-cmd --permanent --add-port=514/tcp # firewall-cmd --permanent --add-port=514/udp # firewall-cmd --reload
You can also limit incoming connections on port 514 from whitelisted IP ranges as shown.
# firewall-cmd --permanent --add-rich-rule 'rule family="ipv4" source address="123.123.123.0/21" port port="514" protocol="tcp" accept' # firewall-cmd --permanent --add-rich-rule 'rule family="ipv4" source address="123.123.123.0/21" port port="514" protocol="udp" accept' # firewall-cmd --reload
That’s all! Rsyslog is now configured as a centralize logs server and can collect logs from remote clients. In the next article, we will see how to configure Rsyslog client on CentOS/RHEL 8 server.
Hi everyone,
Has anyone migrated from CentOS 8 to another Linux flavor since RedHat is pulling the plug on it? Any good documentation developed for it? I THOROUGHLY appreciated this one!!!
@Gator,
I have migrated my CentOS 8 servers to Rocky Linux 8, here is guide – How to Migrate from CentOS 8 to Rocky Linux 8
Hi,
I am trying to forward logs from RHEL 8 server to the RHEL 7 server. As we know the rsyslog format is different. Can anyone suggest, how we can make this compatible that the logs can e forwarded to the lower rhel rsyslog version?
The below error is seen, when logs are forwarded from RHEL 8 to RHEL 7.
Did you get an answer to this? I have the same issue.
My guess is a TLS version mismatch. CentOS 8 uses a newer version.
This article needs a couple of minor revisions.
All references to yum should be replaced with the new CentOS 8 package manager, dnf. Fortunately, since dnf is born of yum, the commands are backwards-compatible. “
yum install xyz” should be replaced with “dnf install xyz“.Minor typo, “
yum whatproviders semanage” should be “dnf whatprovides semanage” (“whatproviders” should be “whatprovides“)". ?RemoteLogs & ~"throws errors in /var/log/messages. It should read"*.* ?RemoteLogs". The"& ~"is no longer needed in the new version of rsyslog shipped by default in CentOS 8.Also, there are major security risks in having a rsyslog server open to the world on port 514. You can expand the article to include examples of using rich rules to limit incoming connections on port 514 from whitelisted IP ranges.
Examples:
# firewall-cmd --permanent --add-rich-rule 'rule family="ipv4" source address="123.123.123.0/21" port port="514" protocol="tcp" accept'
# firewall-cmd --permanent --add-rich-rule 'rule family="ipv4" source address="123.123.123.0/21" port port="514" protocol="udp" accept'
# firewall-cmd --reload
@Corey,
Thanks for the suggestions, corrected in the article…
@Ravi thanks! Just one new type though, please remove the double-quotes surrounding the line:
Should be
@Corey,
Thanks again, corrected the command in the writeup…
Hi
I hope I can get some help with this. I am trying to use rsyslog to collect logs from our firewalls. So I configured the rsyslog.conf file as below.
#### MODULES #### module(load="imuxsock" # provides support for local system logging (e.g. via logger command) SysSock.Use="off") # Turn off message reception via local log socket; # local messages are retrieved through imjournal now. module(load="imjournal" # provides access to the systemd journal StateFile="imjournal.state") # File to store the position in the journal #module(load="imklog") # reads kernel messages (the same are read from journald) #module(load"immark") # provides --MARK-- message capability # Provides UDP syslog reception # for parameters see http://www.rsyslog.com/doc/imudp.html module(load="imudp") # needs to be done just once input(type="imudp" port="514") # TEMPLATE $template RemoteLogs,"/home/logs/%HOSTNAME%/%$YEAR-%$MONTH-%$DAY%/%syslogfacility-text%.log" . ?RemoteLogs & ~Unfortunately, when I check the /home/logs directory I do not see any files or folders created but I see traffic from the firewall going to the CentOS 8 VM. Can anyone help or suggest what I am missing? I have disabled firewalld and SELinux but still cannot get the log files generated.
Template should read.
You are missing some
%. I hope that helps.Finally got it to work. Thank you so much Gator.
Had to redo the template line as:
I think this was necessary to be on its own line also..
The next step after centralizing the logs comes the need to analyze, classify and report on the events. This is where a solution like Graylog beats a rsyslog server hands down.
Also, I was wondering if there was a way to separate the log entries incoming from different servers into separate files or directories (perhaps create a new directory per day)? It would make it easier to perform lookups, archiving, and clean-ups. Thank you, again, for the OUTSTANDING job!!!
@Gator,
You can use the following configuration parameters to write all received messages from clients in a single log file named after the IP Address of the remote client.
I could not get this to work quite right. I wound up doing it a bit differently.
I wanted to split up the logs to be generated into date folders with the clients as subfolders. The reason for this is to make it easy for archiving and cleanup as the directories are all listed by date. I went with the date/client method, but some people may prefer the client/date way in which they would just need to switch around the order of the directory creation.
To do this, I replaced the template lines with:
I then changed it to
to get a better break down of individual process logging. So far it looks good.
Thank you very much again!!!
Hi Ravi,
Thank you for creating a new cookbook for this on CentOS 8.
I did find a few errors that you may want to update:
1. With the basic installation of CentOS, semanage and netstat are not loaded. You want to add “yum whatproviders semanage” and then “yum install policycoreutils-python-utils“.
For netstat, “yum whatproviders netstat” and then “yum install net-tools“.
2. The semanage lines are incorrect. They should read “semanage port -a -t syslogd_port_t -p udp 514” and “semanage port -a -t syslogd_port_t -p tcp 514“.
3. One of the firewall commands is incorrect. It should have 2 dashes: “firewall_cmd –reload“.
Excellent job and very much thanks!!!!
@Gator,
I have corrected everything as suggested by you in the article. Please check and confirm.
Thanks again…:)
It looks spot on. I know a lot of people will find your article very helpful!!! Thank you VERY much!!!