Graylog 4.x on RHEL 8 for Log Monitoring

Graylog 4.x on RHEL 8 for Log Monitoring

"Never underestimate the value of a good set of log files." this was drilled into me a long time ago, and even today a well setup Syslog environment will provide you with far richer information out of the box than a red/amber/green monitoring solution.

Used correctly log files will tell you everything about a system. However, if you have a big server farm or even a decent-sized homelab trying to hop on each server and read those logs is a time zap. Logs are also not just for problems and contain much of the rich data about your environment you need to know...

Over the weekend I've migrated off a cloud-based Syslog aggregator and onto a self-hosted one. Mainly due to costs.

Going from

I've been using Solarwinds PaperTrail over the last few years

Papertrail - cloud-hosted log management, live in seconds
Frustration-free log management. Seamlessly manage logs from apps, servers, and cloud services.

As SaaS-based services go I've got nothing to complain about, this was really just a consolidation costs at my end, those £7.99 here £4.99 there a month charges were mounting up to over £100 a month in SaaS and other services so over Easter I had a look at what I could self-host and what I couldn't, unfortunately, PaperTrail was one of the items that got the chop.  

The interface is based on a nice real-time display of logs collected from systems. This can be filtered on. Adding systems is a curl that pulls down a script and adds in most (Linux) cases an entry to rsyslog.d which in turn forwards all the logs to PAPERTRAIL. Curl is Solar winds method not mine.

Windows, Mac and BSD are also all supported

Once the data is in as you'd expect with Solarwinds products you can filter, setup notifications to a lot of endpoints and generally its a well thought out simple system and one I'll be coming back to at some point I hope..

Going to

I had a look around and Graylog still stands out as a well-supported Syslog aggregator which serves my purposes. I did have a look at Loki and I can see it being the future of Syslog and an interesting use of Grafana. If you know what you're pulling out of logging, then promtail and Loki are possibly an option for you.

For me, it was Graylog 4 Open Source running on RHEL8 with Elasticsearch and MongoDB Back end.

Setup

Install

The most up to date install instructions for installing on CenrOS8/RHEL8 can be found here.

CentOS installation — Graylog 4.0.0 documentation

They are broken down into

Prerequisites

sudo yum install java-1.8.0-openjdk-headless.x86_64 pwgen

MongoDB

Add the repository file  

vi /etc/yum.repos.d/mongodb-org.repo

with the following contents:

[mongodb-org-4.2] 
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/4.2/x86_64/ 
gpgcheck=1 enabled=1 
gpgkey=https://www.mongodb.org/static/pgp/server-4.2.asc

Install MongoDb

sudo yum install mongodb-org

Additionally, run these last steps to start MongoDB during the operating system’s boot and start it right away:

sudo systemctl daemon-reload 
sudo systemctl enable mongod.service 
sudo systemctl start mongod.service 
sudo systemctl --type=service --state=active | grep mongod

Elasticsearch

Graylog can be used with Elasticsearch 7.x,

First, install the Elastic GPG key

sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

then add the repository file

vi /etc/yum.repos.d/elasticsearch.repo 

with the following contents:

[elasticsearch-7.x]
name=Elasticsearch repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/oss-7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

followed by the installation of the latest release

 sudo yum install elasticsearch-oss

Modify the Elasticsearch configuration file

/etc/elasticsearch/elasticsearch.yml

set the cluster name to graylog and uncomment action.auto_create_index: false to enable the action:

sudo tee -a /etc/elasticsearch/elasticsearch.yml > /dev/null <<EOT
cluster.name: graylog 
action.auto_create_index: false 
EOT

After you have modified the configuration, you can start Elasticsearch:

sudo systemctl daemon-reload 
sudo systemctl enable elasticsearch.service 
sudo systemctl restart elasticsearch.service 
sudo systemctl --type=service --state=active | grep elasticsearch

Graylog 4

Now install the Graylog repository configuration and Graylog itself with the following commands:

sudo rpm -Uvh https://packages.graylog2.org/repo/packages/graylog-4.0-repository_latest.rpm 
Note: even though the repo server is graylog2.org the packages are graylog-4.0

Install Graylog

sudo yum install graylog-server graylog-integrations-plugins

Open Ports

sudo firewall-cmd --add-port=9000/tcp --permanent
sudo firewall-cmd --reload

Edit the Graylog Config File

To have Graylog startup some secrets need to be added to the config file found in

/etc/graylog/server/server.conf

Before you open it some secrets need to be created.

Password Salt

DO NOT COPY MY OUTPUTS HERE THEY ARE EXAMPLES

run

 pwgen -N 1 -s 96

Will result in something which looks like this

KUh9nzuKQhO8bx7syIeJrcwlWMwmqMfgzzNYi0wxlCItDbMbSeZWx788o5kDMnZIP08nPiEC3HmpezJyXFbpZJiGTAJSrig9

Password Sha2

DO NOT COPY MY OUTPUTS HERE THEY ARE EXAMPLES

run

echo -n "Enter Password: " && head -1 </dev/stdin | tr -d '\n' | sha256sum | cut -d" " -f1

When prompted Enter the password for the Graylog Admin user you would like to use

Enter Password: MySecr3tP455W0rD

This will generate

9c1b79ce5dc6b962229c81089f86e7154f3f868494f67b13fd4780b68152adef

Edit Graylog Server Config

Now we have these open the Graylog server config

vi /etc/graylog/server/server.conf

Find the section

password_secret = 

and change it to use your salt

password_secret = KUh9nzuKQhO8bx7syIeJrcwlWMwmqMfgzzNYi0wxlCItDbMbSeZWx788o5kDMnZIP08nPiEC3HmpezJyXFbpZJiGTAJSrig9

Find the section

root_password_sha2 = 

and change it to the password sha2 you created

root_password_sha2 = 9c1b79ce5dc6b962229c81089f86e7154f3f868494f67b13fd4780b68152adef

Save the file, but don't exit

There are a lot of settings in here and i'd recommend 2 things

1) Have a scan through them for example you might not want the admin user to be admin

2) Back up this file..

Email Server

If you want to send out mail notifications from Graylog then the /etc/graylog/server/server.conf is where the mail SMTP settings are setup.

I have an internal mail relay on my home Lan which is only accessible by devices on my network. I (currently) don't run SSL/TLS and it uses standard port 25

For this basic setup I've done the following and disabled TLS and SSL

Use SMTP with STARTTLS, see https://en.wikipedia.org/wiki/Opportunistic_TLS
transport_email_use_tls = false

Use SMTP over SSL (SMTPS), see https://en.wikipedia.org/wiki/SMTPS
This is deprecated on most SMTP services!
transport_email_use_ssl = false

then set the following mail entries

transport_email_enabled = true
transport_email_hostname = 192.168.10.132
transport_email_port = 25
transport_email_subject_prefix = [graylog]

Start the Graylog server

sudo systemctl daemon-reload 
sudo systemctl enable graylog-server.service 
sudo systemctl start graylog-server.service 
sudo systemctl --type=service --state=active | grep graylog

If you're using SELinux then the following should be followed or you won't receive any messages.

SELinux information

Hint

We assume that you have policycoreutils-python installed to manage SELinux.

If you’re using SELinux on your system, you need to take care of the following settings:

  • Allow the web server to access the network: sudo setsebool -P httpd_can_network_connect 1
  • If the policy above does not comply with your security policy, you can also allow access to each port individually:
  • Graylog REST API and web interface: sudo semanage port -a -t http_port_t -p tcp 9000
  • Elasticsearch (only if the HTTP API is being used): sudo semanage port -a -t http_port_t -p tcp 9200
  • Allow using MongoDB’s default port (27017/tcp): sudo semanage port -a -t mongod_port_t -p tcp 27017

If you run a single server environment with NGINX or Apache proxy, enabling the Graylog REST API is enough. All other rules are only required in a multi-node setup. Having SELinux disabled during installation and enabling it later, requires you to manually check the policies for MongoDB, Elasticsearch and Graylog.

Hint

Depending on your actual setup and configuration, you might need to add more SELinux rules to get to a running setup.

Open Graylog Web Interface

Open the web interface in your browser

http://<your domain or server ip>:9000

Login with the default login name of

admin

and the password you setup earlier and added the sha2 to the config file.

Graylog will take you to the setup (old screen shot)

Add Ingress

Graylog needs setting up so it can receive log files from remote servers this is done quickly in the web interface

Ignore the Getting Started screen and Click on System -> Inputs

On the Inputs screen Select Syslog TCP from the drop-down menu and then click on Launch new input

You'll be asked for some information, this example will display the basics required to get syslog over TCP into Graylog

Use port 1514/TCP because server is started as a user and can't use ports below 1024 so if you try and use 514/TCP it will fail with an error which doesn't tell you much.

I did tick

  • Allow overriding date?
  • Store fill message?
  • Expand structured data ?

Click on Save

Start the Ingress local input

On the graylog server, the port 1514/tcp needs opening on the firewall

sudo firewall-cmd --add-port=1514/tcp --permanent
sudo firewall-cmd --reload

Your server is now ready to receive Syslog from rsyslog on Linux servers.

Setting up rsyslog to send logs from a remote server

Manually setting up remote servers is pretty simple if you're using rsyslogd

edit a .d config file in

sudo vi /etc/rsyslog.d/90-graylog.conf
You can call the file what you want but it must end in .conf

add the line

*.* @@<graylog server ip>:1514;RSYSLOG_SyslogProtocol23Format

so as an example

*.* @@192.168.10.141:1514;RSYSLOG_SyslogProtocol23Format

and restart rsyslog

service rsyslog restart
service rsyslog status

It's worth noting that SELinux may throw errors when you run the status command, if it does run

sudo semanage port -a -t syslogd_port_t -p tcp 1514

You should start seeing logs flow into your server after a few minutes depending on how busy your servers are.

Ansible to Update Servers

If you've got more than 2 or 3 devices then using ansible its fairly quick yo run something like the following

- hosts: all
  tasks:
  - name: Creating a file with content
    copy:
      dest: "/etc/rsyslog.d/90-graylog.conf"
      content: |
        *.* @@192.168.10.141:1514;RSYSLOG_SyslogProtocol23Format

  - name: Start restart rsyslogd service
    ansible.builtin.systemd:
      state: restarted
      daemon_reload: yes
      name: rsyslog

I've run over 12 Red Hat 8 servers and worked fine.

Setup summary

At this point, Graylog 4 is set up on a single server listening for logs on 1514/tcp and we have set up the remote servers to send their logs to the Graylog server.

Post Setup

Searching

Searching — Graylog 4.0.0 documentation

The core of any such solution is being able to search through your log files and this is the base of what Graylog offers

Click on Search

You'll be presented with a search screen that will by default display all the logs in a static view for all the servers pushing syslog data into Graylog

This is a snapshot in time and you can set this screen to update

Clicking on the Update (play button) will refresh to screen accordingly and provide a true realtime view.

There is a whole structured querying language behind Graylog to pull data out of your logs in the example above i've  set the search to look at the logs from only one server

source: mediaserver

Understanding how to search is important for the next two sections Dashboards and Notifications.

Search query language — Graylog 4.0.0 documentation

Dashboards

Dashboards — Graylog 4.0.0 documentation

Dashboards are a method of displaying data from searches in Charts, Tables or counters on a predefined screen.

Clicking on the + icon having created a dashboard will provide the basic Count and Table options

Clicking on Message Count as an example provides an interface where you can define a search to narrow down the count. Create a title and customize the counter.

However, with a little reading its possible to create some really pretty dashboards

Notifications

If there are log entries you need to know when they pop up on the servers, Alerts can be created

Alerts — Graylog 4.0.0 documentation

The Link here will go through setting up alerts and link them to events and notifications.

In my test, I have set up an alert when something sends out of my mail relay.

Something to remember however if you choose to use email as the notification method. If it's not setup in /etc/graylog/server/server.conf as was shown earlier, you will get a myriad of errors that won't strictly help you resolve the issues.

Check the mail log of the relay as well, as it may be connecting on 25 but failing because it's trying on SSL like mine was.

Lots More

There is so so so much more Graylog can do, this post is my initial setup "guide" which gets the basics up and running, in a proper prod environment the service can be setup as HA, a Reverse proxy could be put in front of this setup to provide https access.

There is authentication to consider so the login is via AD or LDAP then setting up teams and groups, there's ingressing Windows logs

There are content packs for expanding how logss are looked for and pulling out known security exploits

So much to do...

Thoughts

There is a lot to the OSS version of Graylog 4, my next step is to write an Ansible Playbook using the instructions above to install the server using ansible.


Share Tweet Send
0 Comments
Loading...
You've successfully subscribed to Tech Blog Posts - David Field
Great! Next, complete checkout for full access to Tech Blog Posts - David Field
Welcome back! You've successfully signed in
Success! Your account is fully activated, you now have access to all content.