SensuGO, Installation and Basic Monitoring Setup

14 min read
SensuGO, Installation and Basic Monitoring Setup

The following post is written to take you from Zero to Sensu Go providing some basic monitoring without the need for all the code. As I work through learning how Sensu works with Monitoring as Code deployments I will write a follow-up post on how to do Monitoring a code.

Version Date Description Notes
1.0 21 Dec 2021 First Draft Probably plenty of spelling errors

What is it

Sensu | Get Started with Sensu Go
Try Sensu Go with our quick-start guide! Get Sensu Go up and running in 60 seconds or less.

Sensu is a complete solution for monitoring and observability at scale. Sensu Go is designed to give you visibility into everything you care about: traditional server closets, containers, applications, the cloud, and more.

Sensu is an agent-based observability tool that you install on your organization’s infrastructure. The Sensu backend gives you a flexible, automated pipeline to filter, transform, and process alerts and metrics.

Sensu Go is operator-focused and developer-friendly and integrates with popular monitoring and observability tools. Deploy Sensu Go for on-premises and public cloud infrastructures, containers, bare metal, or any other environment.

Above all Sensu Go is Monitoring as Code.

Setup

Operating System Redhat 8
vCPU 4
RAM 4Gb
HDD 320Gb
Hosting environment Proxmox

Installation

The installation is pretty straightforward and easily scriptable

Open Firewall Ports

Sensu backends require the following ports:

2379gRPCSensu storage client: Required for Sensu backends using an external etcd instance
2380gRPCSensu storage peer: Required for other Sensu backends in a cluster
3000HTTP/HTTPSSensu web UI: Required for all Sensu backends using a Sensu web UI
6060HTTP/HTTPSRequired for all Sensu backends when performance profiling is enabled via debug setting
8080HTTP/HTTPSSensu API: Required for all users accessing the Sensu API
8081WS/WSSAgent API (backend WebSocket): Required for all Sensu agents connecting to a Sensu backend

The Sensu agent uses the following ports:

3030TCP/UDPSensu agent socket: Required for Sensu agents using the agent socket
3031HTTPSensu agent API: Required for all users accessing the agent API
8125UDPStatsD listener: Required for all Sensu agents using the StatsD listener

The agent TCP and UDP sockets are deprecated in favor of the agent API.

Install the Sensu backend

The Sensu backend is available for Ubuntu/Debian, RHEL/CentOS, and Docker. Review supported platforms for more information.

RHEL/CentOS

# Add the Sensu repository
curl -s https://packagecloud.io/install/repositories/sensu/stable/script.rpm.sh | sudo bash

# Install the sensu-go-backend package
sudo yum install sensu-go-backend

Configure and start

You can configure the Sensu backend with sensu-backend start flags (recommended) or an /etc/sensu/backend.yml file. The Sensu backend requires the state-dir flag at minimum, but other useful configurations and templates are available.

RHEL/CentOS

# Copy the config template from the docs
sudo curl -L https://docs.sensu.io/sensu-go/latest/files/backend.yml -o /etc/sensu/backend.yml

# Start sensu-backend using a service manager
sudo service sensu-backend start

# Verify that the backend is running
service sensu-backend status

The backend reference includes a complete list of configuration options and backend initialization details.

WARNING: If you plan to run a Sensu cluster, make sure that each of your backend nodes is configured, running, and a member of the cluster before you continue the installation process.

Initialize

NOTE: If you are using Docker, you already completed intitialization in 2. Configure and start. Skip ahead to 4. Open the web UI to continue the backend installation process. If you did not use environment variables to override the default admin credentials in step 2, skip ahead to Install sensuctl so you can change your default admin password immediately.

With the backend running, run sensu-backend init to set up your Sensu administrator username and password. In this initialization step, you only need to set environment variables with a username and password string — no need for role-based access control (RBAC).

Replace <username> and <password> with the username and password you want to use:Ubuntu/DebianRHEL/CentOS

export SENSU_BACKEND_CLUSTER_ADMIN_USERNAME=<username>
export SENSU_BACKEND_CLUSTER_ADMIN_PASSWORD=<password>
sensu-backend init

For details about initializing the Sensu backend, read the backend reference.

NOTE: You may need to allow access to the ports Sensu requires in your local server firewall. Refer to the documentation for your operating system to configure port access as needed.

Open the web UI

The web UI provides a unified view of your observability events and user-friendly tools to reduce alert fatigue. After starting the Sensu backend, open the web UI by visiting http://localhost:3000. You may need to replace localhost with the hostname or IP address where the Sensu backend is running.

To log in to the web UI, enter your Sensu user credentials.  Your user credentials are the username and password you provided with the SENSU_BACKEND_CLUSTER_ADMIN_USERNAME and SENSU_BACKEND_CLUSTER_ADMIN_PASSWORD environment variables.

Install sensuctl

Sensuctl is a command line tool for managing resources within Sensu. It works by calling Sensu’s HTTP API to create, read, update, and delete resources, events, and entities. Sensuctl is available for Linux, Windows, and macOS.

RHEL/CentOS

# Add the Sensu repository
curl https://packagecloud.io/install/repositories/sensu/stable/script.rpm.sh | sudo bash

# Install the sensu-go-cli package
sudo yum install sensu-go-cli

To start using sensuctl, run sensuctl configure and log in with your user credentials, namespace, and Sensu backend URL. To configure sensuctl using default values:

sensuctl configure -n \
--username 'YOUR_USERNAME' \
--password 'YOUR_PASSWORD' \
--namespace default \
--url 'http://127.0.0.1:8080'

Here, the -n flag triggers non-interactive mode. Run sensuctl config view to view your user profile.

For more information about sensuctl, read the sensuctl documentation.

Change default admin password

If you are using Docker and you did not use environment variables to override the default admin credentials in step 2 of the backend installation process, we recommend that you change the default admin password as soon as you have installed sensuctl. Run:

sensuctl user change-password --interactive

Install Sensu agents

The Sensu agent is available for Ubuntu/Debian, RHEL/CentOS, Windows, and Docker. Review supported platforms for more information.

RHEL/CentOS

# Add the Sensu repository
curl -s https://packagecloud.io/install/repositories/sensu/stable/script.rpm.sh | sudo bash

# Install the sensu-go-agent package
sudo yum install sensu-go-agent

Configure and start

You can configure the Sensu agent with sensu-agent start flags (recommended) or an /etc/sensu/agent.yml file. The Sensu agent requires the --backend-url flag at minimum, but other useful configurations and templates are available.

RHEL/CentOS

# Copy the config template from the docs
sudo curl -L https://docs.sensu.io/sensu-go/latest/files/agent.yml -o /etc/sensu/agent.yml

This file needs to be edited, there are a lot of items in the YAML file where additional security can be added, certificates used and much more, I'd strongly suggest ensuring this is set up as you'd expect for a production environment.

The agent reference includes a complete list of configuration options.

/etc/sensu/agent.yml

The following fields need to be changed as a minimum

name: <- name of the host being monitored, this is how it will display in Sensu
labels: <- Useful for grouping
backend-url: <- URL of the Sensu Server

Example

name: "gitserver.lan"
labels:
  os: "rhel8"
  app: "git"
backend-url:
  - "ws://192.168.40.151:8081"

Start the service

# Start sensu-agent using a service manager
systemctl start sensu-agent
systemctl enable sensu-agent
systemctl status sensu-agent

Verify keepalive events

Sensu keepalives are the heartbeat mechanism used to ensure that all registered agents are operating and can reach the Sensu backend. To confirm that the agent is registered with Sensu and is sending keepalive events, open the entity page in the Sensu web UI or run sensuctl entity list.

Monitoring using the Gui

Sensu is all about Monitoring as code, I'm however all about getting something working as quickly so before I delved into the world of Monitoring as Code I wanted to get something in place quickly to make sure that Sensu was at least something I could use.

Monitor server resources with checks | Sensu Go 6.6
Sensu lets you monitor server resources with checks. Read this guide to learn about Sensu checks and how to use checks to monitor servers.

Checking the agents are communicating

Logging onto the Sensu web interface, the default dashboard is presented.

Click on the entities link

A list of monitored devices should appear with the basic keepalive checks.

The next stage is to add some assets to Sensu and start monitoring.

Sensu Assets

Sensu assets | Bonsai asset index - Assets Directory
Welcome to Bonsai, the Sensu asset index. Bonsai offers popular integrations like the Sensu Slack, InfluxDB, and PagerDuty handlers. Discover, download, and share assets for all your monitoring needs.

Assets, or code that performs specific monitoring tasks are held at Bonsai. there are also Assets on Github. Bonsai at least it a safe location to search.

I found there to be two levels of the asset on Bonsai for a beginner, the well defined, documented and generally working kind, and the thrown together, with assumed knowledge kind. I'm sure as I work through understanding the software this will become more clear.

An example of a well-documented asset looks like this.

Adding an Asset from Bonsai

The simplest method I've found for adding an asset from Bonsai is using the command line.

Example:

sensuctl asset add sensu/sensu-influxdb-handler:3.7.0

Armed with this, the following section will walk through how to setup 4 different checks in Sensu using the WebGui.

Add Asset

CPU

sensuctl asset add check-cpu-usage

RAM

sensuctl asset add asachs01/sensu-go-memory-check

Disk Space

sensuctl asset add

HTTP 200 Response

sensuctl asset add

To view the assets installed run

sensuctl asset list

Each asset should have a tar.gz for Linux, Windows, OSX at a minimum

  asachs01/sensu-go-memory-check               //assets.bonsai.sensu.io/.../sensu-go-memory-check_1.0.1_linux_arm64.tar.gz
  
  asachs01/sensu-go-memory-check               //assets.bonsai.sensu.io/.../sensu-go-memory-check_1.0.1_linux_386.tar.gz
  
  asachs01/sensu-go-memory-check               //assets.bonsai.sensu.io/.../sensu-go-memory-check_1.0.1_linux_amd64.tar.gz
  
  asachs01/sensu-go-memory-check               //assets.bonsai.sensu.io/.../sensu-go-memory-check_1.0.1_windows_amd64.tar.gz
  
  asachs01/sensu-go-memory-check               //assets.bonsai.sensu.io/.../sensu-go-memory-check_1.0.1_darwin_386.tar.gz   
  
  asachs01/sensu-go-memory-check               //assets.bonsai.sensu.io/.../sensu-go-memory-check_1.0.1_darwin_amd64.tar.gz 
  
  asachs01/sensu-go-memory-check               //assets.bonsai.sensu.io/.../sensu-go-memory-check_1.0.1_linux_armv7.tar.gz

Setup an asset to monitor a Host

With the assets installed, the checks can be put in place for the hosts we are monitoring.

I will note again when using Sensu in production, the purpose of the software is to have this done and rolled out as code. What I'm showing here is the quick simple, manual method to allow you to see how Sensu works and get up and running quickly.

Adding a Manual Check

Manual checks are added Via the Entities section of the webportal

Within the Entities section click on the hamburger menu to the left and click on Checks

In the top right of the check screen click on + New

A dialogue box will pop up

The following sections will be needed to create a basic check.

Execution

The execution is the command which will be run, remember that you've got an agent on the host you are monitoring, so the check will be pushed out to the endpoint and run there locally NOT from the Sensu server.

The command can (if the docs are well written) be found with any switches on the Bonsai page under Usage Examples

Clicking on assets will generate a drop-down list of available assets installed using the sensuctl asset add command earlier

Target

The Target is where the check should be run, this can be either the entity name

Using this method it's possible to push a check out to a group, groups, individual hosts or a mix of all three.

Schedule

When to run the check, the timing is handled in seconds. the scheduling itself can be done either using Sensu OR kicking off a cron job.

Result Handling

This will be covered at the end of the post, I have all my alerts go into a Discord channel, they could, however, go to email, SMS, slack etc or multiple locations based on frequency.

Setting up Checks

CPU

Execution Command check-cpu-usage -w 85 -c 95
Execution Asset check-cpu-usage
Target Subscriptions Select Group or host
Schedule How Internal Scheduling
Schedule Interval 300s

RAM

Execution Command sensu-go-memory-check -w 85 -c 90
Execution Asset asachs01/sensu-go-memory-check
Target Subscriptions Select Group or host
Schedule How Internal Scheduling
Schedule Interval 120

Disk Space

Execution Command check-disk-usage --exclude-fs-path "/boot" --warning 90 --critical 95 --human-readable
Execution Asset sensu/check-disk-usage
Target Subscriptions Select Group or host
Schedule How Internal Scheduling
Schedule Interval 600

HTTP 200 Response

Execution Command http-check --url http://localhost
Execution Asset sensu/http-checks
Target Subscriptions Select Group or host
Schedule How Internal Scheduling
Schedule Interval 300

Once the check is added Sensu will go and run the check on the remote host.

Viewing the Checks

A good place to view the checks, the data from them and other information is via the entities view

Select a host to view the status of the checks running on it.

Click on a check to find out more about its status

Notifications to Discord

As mentioned previously, the alerts on my home system are sent to a private discord channel using the discord web API

I used the handler found here

GitHub - runarsf/sensugo-discord-handler: Get status updates from SensuGo directly in Discord!
Get status updates from SensuGo directly in Discord! - GitHub - runarsf/sensugo-discord-handler: Get status updates from SensuGo directly in Discord!

To install it, on the Sensu server do the following.

Clone the repository

git clone https://github.com/runarsf/sensugo-discord-handler.git

Add it to path (in this example /usr/local/bin)

ln -s $(pwd)/sensugo-discord-handler.py /usr/local/bin/sensugo-discord-handler

Create a Discord Webhook

  • Open Discord (Desktop / Web, not currently available on the mobile app) and navigate to the channel you want the status messages to be posted in
  • Click Edit Channel > Webhooks > Create Webhook
  • Name the webhook something recognizable, the name will only matter when managing it, as the name visible in chat will be provided by the script.
  • The same goes for the icon, the script will provide a custom one.
  • Take note of the WEBHOOK URL, you will need it later (you can view it at any time).

Modify handler.json file to match your setup.

{
  "type": "Handler",
  "api_version": "core/v2",
  "metadata": {
    "name": "discord",
    "namespace": "default"
  },
  "spec": {
    "command": "/usr/bin/sensugo-discord-handler",
    "env_vars": [
          "WEBHOOK_URL=https://discord.com/api/webhooks/906265893808533554/573US96OnPs9AT4x_bwzKWR91U1dfdsjdafhlaf;afsipuafsZF7HNGUL5",
       "USERNAME=SensuGo",
       "ICON_URL=https://docs.sensu.io/images/sensu-logo-icon-dark@2x.png",
  "USE_EMBED=True"
],
"filters": [
  "is_incident",
  "not_silenced",
  "state_change_only"
],
"handlers": [],
"runtime_assets": [],
"timeout": 0,
"type": "pipe"
   }
 } 

Create the Handler from the file.

sensuctl create --file handler.json 

You should now have a new handler in your SensuGo WebUI

View the Handler

On the Sensu WebUI

The details of the Handler should be visible

This handler can then be added to any check

Note

In the handler.yaml there is a filter section

"filters": [
  "is_incident",
  "not_silenced",
  "state_change_only"
],

I have found if i want all alerts I actually just use

"filters": [
  "is_incident",
],

Thoughts

There is enough here to get you up and running with Sensu to evaluate the WebUI and monitoring methodology and see if this is for you.

Sensu is very different from the systems I've previously used like Opsview, GroundWork and checkMK in its presentation, and I think has a lot of possibilities using it as code.

I know the core strength is the ability to integrate this with a build through Jenkins or Rundeck and have the devices monitoring managed with code and that is the next step for me.

I've been running this for a couple of months as a replacement for checkMK, its a far lighter solution, most of the basics are pretty easy to get set up, in addition to the above things like checking services are running, checking sites have a 200 response code that sort of thing.

I think for a beginner some of the items on bonsai are very light on usable instructions. however, I'm hoping as I learn more about this it will become clearer.

References

Managing Sensu Go 6 using Ansible | XLAB Steampunk blog
This post was originally published on the Sensu Go blog.Earlier this year, we shared the certified Ansible Collection for Sensu Go, which makes it easy to automate your monitoring and achieve real-time visibility into auto-scaling infrastructure. Now that Sensu Go 6 has been released, we will share…
Sensu Go — A complete installation guide (Monitoring mission-critical systems)
Are you looking to set up a real-time efficient monitoring tool for your business? then don’t miss out on reading this blog. Monitoring…
Monitoring as code with Sensu + Ansible - Stack Over Cloud
A comprehensive infrastructure as code (IaC) initiative should include monitoring and observability. Incorporating the active monitoring of the infrastructure under management results in a symbiotic relationship in which failures are detected automatically, enabling event-driven code changes and new…
GitHub - runarsf/sensugo-discord-handler: Get status updates from SensuGo directly in Discord!
Get status updates from SensuGo directly in Discord! - GitHub - runarsf/sensugo-discord-handler: Get status updates from SensuGo directly in Discord!
Tech Blog Posts - David Field



Follow