“Never underestimate the value of a good set of log files.” this was drilled into me a long time ago, and even today a well setup Syslog environment will provide you with far richer information out of the box than a red/amber/green monitoring solution.
Used correctly log files will tell you everything about a system. However, if you have a big server farm or even a decent-sized homelab trying to hop on each server and read those logs is a time zap. Logs are also not just for problems and contain much of the rich data about your environment you need to know…
Over the weekend I’ve migrated off a cloud-based Syslog aggregator and onto a self-hosted one. Mainly due to costs.
I’ve been using Solarwinds PaperTrail over the last few years
As SaaS-based services go I’ve got nothing to complain about, this was really just a consolidation costs at my end, those £7.99 here £4.99 there a month charges were mounting up to over £100 a month in SaaS and other services so over Easter I had a look at what I could self-host and what I couldn’t, unfortunately, PaperTrail was one of the items that got the chop.
The interface is based on a nice real-time display of logs collected from systems. This can be filtered on. Adding systems is a curl that pulls down a script and adds in most (Linux) cases an entry to rsyslog.d which in turn forwards all the logs to PAPERTRAIL. Curl is Solar winds method not mine.
Windows, Mac and BSD are also all supported
Once the data is in as you’d expect with Solarwinds products you can filter, setup notifications to a lot of endpoints and generally its a well thought out simple system and one I’ll be coming back to at some point I hope..
I had a look around and Graylog still stands out as a well-supported Syslog aggregator which serves my purposes. I did have a look at Loki and I can see it being the future of Syslog and an interesting use of Grafana. If you know what you’re pulling out of logging, then promtail and Loki are possibly an option for you.
For me, it was Graylog 4 Open Source running on RHEL8 with Elasticsearch and MongoDB Back end.
The most up to date install instructions for installing on CenrOS8/RHEL8 can be found here.
They are broken down into
sudo yum install java-1.8.0-openjdk-headless.x86_64 pwgen
Add the repository file
with the following contents:
[mongodb-org-4.2] name=MongoDB Repositorybaseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/4.2/x86_64/ gpgcheck=1 enabled=1 gpgkey=https://www.mongodb.org/static/pgp/server-4.2.asc
sudo yum install mongodb-org
Additionally, run these last steps to start MongoDB during the operating system’s boot and start it right away:
sudo systemctl daemon-reload sudo systemctl enable mongod.service sudo systemctl start mongod.service sudo systemctl --type=service --state=active | grep mongod
Graylog can be used with Elasticsearch 7.x,
First, install the Elastic GPG key
sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
then add the repository file
with the following contents:
[elasticsearch-7.x]name=Elasticsearch repository for 7.x packagesbaseurl=https://artifacts.elastic.co/packages/oss-7.x/yumgpgcheck=1gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearchenabled=1autorefresh=1type=rpm-md
followed by the installation of the latest release
sudo yum install elasticsearch-oss
Modify the Elasticsearch configuration file
set the cluster name to
graylog and uncomment
action.auto_create_index: false to enable the action:
sudo tee -a /etc/elasticsearch/elasticsearch.yml > /dev/null <<EOTcluster.name: graylog action.auto_create_index: false EOT
After you have modified the configuration, you can start Elasticsearch:
sudo systemctl daemon-reload sudo systemctl enable elasticsearch.service sudo systemctl restart elasticsearch.service sudo systemctl --type=service --state=active | grep elasticsearch
Now install the Graylog repository configuration and Graylog itself with the following commands:
sudo rpm -Uvh https://packages.graylog2.org/repo/packages/graylog-4.0-repository_latest.rpm
Note: even though the repo server is graylog2.org the packages are graylog-4.0
sudo yum install graylog-server graylog-integrations-plugins
sudo firewall-cmd --add-port=9000/tcp --permanent
sudo firewall-cmd --reload
Edit the Graylog Config File
To have Graylog startup some secrets need to be added to the config file found in
Before you open it some secrets need to be created.
DO NOT COPY MY OUTPUTS HERE THEY ARE EXAMPLES
pwgen -N 1 -s 96
Will result in something which looks like this
DO NOT COPY MY OUTPUTS HERE THEY ARE EXAMPLES
echo -n "Enter Password: " && head -1 </dev/stdin | tr -d '\n' | sha256sum | cut -d" " -f1
When prompted Enter the password for the Graylog Admin user you would like to use
Enter Password: MySecr3tP455W0rD
This will generate
Edit Graylog Server Config
Now we have these open the Graylog server config
Find the section
and change it to use your salt
password_secret = KUh9nzuKQhO8bx7syIeJrcwlWMwmqMfgzzNYi0wxlCItDbMbSeZWx788o5kDMnZIP08nPiEC3HmpezJyXFbpZJiGTAJSrig9
Find the section
and change it to the password sha2 you created
root_password_sha2 = 9c1b79ce5dc6b962229c81089f86e7154f3f868494f67b13fd4780b68152adef
Save the file, but don’t exit
There are a lot of settings in here and i’d recommend 2 things
1) Have a scan through them for example you might not want the admin user to be admin
2) Back up this file..
If you want to send out mail notifications from Graylog then the /etc/graylog/server/server.conf is where the mail SMTP settings are setup.
I have an internal mail relay on my home Lan which is only accessible by devices on my network. I (currently) don’t run SSL/TLS and it uses standard port 25
For this basic setup I’ve done the following and disabled TLS and SSL
Use SMTP with STARTTLS, see https://en.wikipedia.org/wiki/Opportunistic_TLStransport_email_use_tls = falseUse SMTP over SSL (SMTPS), see https://en.wikipedia.org/wiki/SMTPSThis is deprecated on most SMTP services!transport_email_use_ssl = false
then set the following mail entries
transport_email_enabled = truetransport_email_hostname = 192.168.10.132transport_email_port = 25transport_email_subject_prefix = [graylog]
Start the Graylog server
sudo systemctl daemon-reload sudo systemctl enable graylog-server.service sudo systemctl start graylog-server.service sudo systemctl --type=service --state=active | grep graylog
If you’re using SELinux then the following should be followed or you won’t receive any messages.
We assume that you have
policycoreutils-pythoninstalled to manage SELinux.
If you’re using SELinux on your system, you need to take care of the following settings:
- Allow the web server to access the network:
sudo setsebool -P httpd_can_network_connect 1
- If the policy above does not comply with your security policy, you can also allow access to each port individually:
- Graylog REST API and web interface:
sudo semanage port -a -t http_port_t -p tcp 9000
- Elasticsearch (only if the HTTP API is being used):
sudo semanage port -a -t http_port_t -p tcp 9200
- Allow using MongoDB’s default port (27017/tcp):
sudo semanage port -a -t mongod_port_t -p tcp 27017
If you run a single server environment with NGINX or Apache proxy, enabling the Graylog REST API is enough. All other rules are only required in a multi-node setup. Having SELinux disabled during installation and enabling it later, requires you to manually check the policies for MongoDB, Elasticsearch and Graylog.
Depending on your actual setup and configuration, you might need to add more SELinux rules to get to a running setup.
Open Graylog Web Interface
Open the web interface in your browser
http://<your domain or server ip>:9000
Login with the default login name of
and the password you setup earlier and added the sha2 to the config file.
Graylog will take you to the setup (old screen shot)
Graylog needs setting up so it can receive log files from remote servers this is done quickly in the web interface
Ignore the Getting Started screen and Click on System -> Inputs
On the Inputs screen Select Syslog TCP from the drop-down menu and then click on Launch new input
You’ll be asked for some information, this example will display the basics required to get syslog over TCP into Graylog
Use port 1514/TCP because server is started as a user and can’t use ports below 1024 so if you try and use 514/TCP it will fail with an error which doesn’t tell you much.
I did tick
- Allow overriding date?
- Store fill message?
- Expand structured data ?
Click on Save
Start the Ingress local input
On the graylog server, the port 1514/tcp needs opening on the firewall
sudo firewall-cmd --add-port=1514/tcp --permanent
sudo firewall-cmd --reload
Your server is now ready to receive Syslog from rsyslog on Linux servers.
Setting up rsyslog to send logs from a remote server
Manually setting up remote servers is pretty simple if you’re using rsyslogd
edit a .d config file in
sudo vi /etc/rsyslog.d/90-graylog.conf
You can call the file what you want but it must end in .conf
add the line
*.* @@<graylog server ip>:1514;RSYSLOG_SyslogProtocol23Format
so as an example
and restart rsyslog
service rsyslog restartservice rsyslog status
It’s worth noting that SELinux may throw errors when you run the status command, if it does run
sudo semanage port -a -t syslogd_port_t -p tcp 1514
You should start seeing logs flow into your server after a few minutes depending on how busy your servers are.
Ansible to Update Servers
If you’ve got more than 2 or 3 devices then using ansible its fairly quick yo run something like the following
- hosts: all tasks: - name: Creating a file with content copy: dest: "/etc/rsyslog.d/90-graylog.conf" content: | *.* @@192.168.10.141:1514;RSYSLOG_SyslogProtocol23Format - name: Start restart rsyslogd service ansible.builtin.systemd: state: restarted daemon_reload: yes name: rsyslog
I’ve run over 12 Red Hat 8 servers and worked fine.
At this point, Graylog 4 is set up on a single server listening for logs on 1514/tcp and we have set up the remote servers to send their logs to the Graylog server.
The core of any such solution is being able to search through your log files and this is the base of what Graylog offers
Click on Search
You’ll be presented with a search screen that will by default display all the logs in a static view for all the servers pushing syslog data into Graylog
This is a snapshot in time and you can set this screen to update
Clicking on the Update (play button) will refresh to screen accordingly and provide a true realtime view.
There is a whole structured querying language behind Graylog to pull data out of your logs in the example above i’ve set the search to look at the logs from only one server
Understanding how to search is important for the next two sections Dashboards and Notifications.
Dashboards are a method of displaying data from searches in Charts, Tables or counters on a predefined screen.
Clicking on the + icon having created a dashboard will provide the basic Count and Table options
Clicking on Message Count as an example provides an interface where you can define a search to narrow down the count. Create a title and customize the counter.
However, with a little reading its possible to create some really pretty dashboards
If there are log entries you need to know when they pop up on the servers, Alerts can be created
The Link here will go through setting up alerts and link them to events and notifications.
In my test, I have set up an alert when something sends out of my mail relay.
Something to remember however if you choose to use email as the notification method. If it’s not setup in /etc/graylog/server/server.conf as was shown earlier, you will get a myriad of errors that won’t strictly help you resolve the issues.
Check the mail log of the relay as well, as it may be connecting on 25 but failing because it’s trying on SSL like mine was.
There is so so so much more Graylog can do, this post is my initial setup “guide” which gets the basics up and running, in a proper prod environment the service can be setup as HA, a Reverse proxy could be put in front of this setup to provide https access.
There is authentication to consider so the login is via AD or LDAP then setting up teams and groups, there’s ingressing Windows logs
There are content packs for expanding how logss are looked for and pulling out known security exploits
So much to do…
There is a lot to the OSS version of Graylog 4, my next step is to write an Ansible Playbook using the instructions above to install the server using ansible.