Rundeck 3: Install, Setup a Project and Add Nodes

Introduction

Rundeck is both a hugely powerful method of centralising your scripts and time based (cron) jobs and a right royal pain in the bum to setup. Being an open source project its changed over may iterations and in doing so its an example of finding information on the web when problems arise and because its an older version the resolutions are not easy to come by.

As a project it also needs (for my sysadmin rather than developer mind) you to think a little different from similar tools like Jenkins

When setup however, its a hugely powerful project and well worth investigating

Disclaimer

These notes are best used to get your feet wet with Rundeck, i'm using password based SSH authentication in the initial release of this post (which will get updated over time) and id rather be using Keys, i'm not using the database back end and i'm learning as I write this. I'm happy to take measured feedback and update/credit the best place to do this is via reddit.

https://www.reddit.com/user/mightywomble

Environment

I'm running this in the following environment

CentOS 7

4Gb Ram

250Gb HDD

All commands run as root (via sudo)

Install

Setup OS

Update the OS

yum update

Disable SELinux

vi /etc/selinux/config

Change

SELINUX=enabled

to

SELINUX=disabled

Reboot your system

reboot

Install Rundeck

Install the latest repo data

rpm -Uvh http://repo.rundeck.org/latest.rpm

Then install the packages

sudo yum install rundeck java rundeck-cli

Open Ports

The Centos Firewall blocks all to start with, you will need to open port 4440/tcp

   firewall-cmd --add-port=4440/tcp --permanent
   firewall-cmd --reload

Edit Settings

The rundeck config files need editing so when you click on links in the Rundeck Gui they work properly remotely

vi /etc/rundeck/rundeck-config.properties 

Change the lines

grails.serverURL=http://localhost:4440

to

grails.serverURL=http://rundeck:4440

then edit the framework file

vi /etc/rundeck/framework.properties 

change

framework.server.name = localhost
framework.server.hostname = localhost
framework.server.port = 4440
framework.server.url = http://localhost:4440

to

framework.server.name = rundeck
framework.server.hostname = rundeck
framework.server.port = 4440
framework.server.url = http://rundeck:4440

Change the Rundeck Login Password

The Default login/password for Rundeck is admin/admin and we should change this.

vi /etc/rundeck/realm.properties

change the line

admin:admin,user,admin,architect,deploy,build

to

admin:YoUrNeWpaSsWoRdHeRe,user,admin,architect,deploy,build

save and exit

Create Projects in the file system not the Database

Note: There are a lot of online instructions which point to the working directory on the file system, however on the version I installed projects data is saved in a Db, while i'm learning the system and little documentation on the Rundeck site at the time of writing has details about the Db side I've changed the install config to create projects in the file system

The location the projects are stored in is /var/lib/rundeck/projects

vi /etc/rundeck/rundeck-config.properties

find the line

rundeck.projectsStorageType=db

change it to

rundeck.projectsStorageType=file

save and exit

Start Rundeck

Enable Rundeck so it starts after a reboot

systemctl enable rundeckd

Start the service

systemctl start rundeckd

Rundeck takes a few minutes to start, and we can check the service log file for indication

tail -f /var/log/rundeck/service.log

If the following line is seen

Grails application running at http://localhost:4440 in environment: production

The server has started

Login

Open Rundeck in the browser

http://rundeck:4440

You cal login using the admin and password you set

This will open the rundeck gui

Setup Project

The core of Rundecks world is projects, projects are the overarching containers for nodes and jobs, you can have multiple nodes or jobs in a Project

Access control to projects can be applied as well, allowing RBAC linked to LDAP/AD to resent users with projects they should have access to.

Create a new project, and give it a name, label and description

Click on Create

Once created the project view in the WebGUI provides a more functional interface

Setup Remote Nodes

Setting up nodes was the first stumbling block I had on this journey as the setup points project config to the DB by default. During the setup above I had the projects be managed as files rather than a database.

Having had projects create as a files and realising /var/lib/rundeck was the root application folder not /var/rundeck as many instructions state. I've found two methods of setting up nodes

SECURITY NOTE: I am aware that ideally this should be done using passwordless ssh keys, and once my setup is working i'll add this option.

However before we use ither of them we need to setup our password in the Rundesck Key Storage

Open the Key Storage page under the Cog to the top right

Select Key Storage

Click on Add Key

Select password from the dropdown

In Enter Text add your password

In name enter the username

Click on Save

Make a note of the storage path

In this example keys/david

Options for adding Nodes

Option 1: Per project using the command line

This option works best if you need different groups of nodes per project

Navigate to your project folder on the Rundeck server

cd /var/lib/rundeck/projects/PROJECTNAME/etc

create a new file resources.xml

vi resources.xml

add using the following example

<?xml version="1.0" encoding="UTF-8"?>
<project>
<node name="homerepo"
  tags="opensusei, linux"
  osFamily="unix"
  username="david"
  hostname="192.168.86.167"
  ssh-authentication="password"
  sudo-command-enabled="true"
  ssh-password-option="option.sshPassword1"
  sudo-command-enabled="true"
  sudo-password-storage-path="keys/david"
  file-copier="ssh-copier"
  ssh-options="-o ConnectTimeout=5000"/>
 />
</project

if you want to add multiple nodes it would look like this

<?xml version="1.0" encoding="UTF-8"?>
<project>
<node name="proxy"
  tags="opensuse, linux"
  osFamily="unix"
  username="david"
  hostname="192.168.20.167"
  ssh-authentication="password"
  sudo-command-enabled="true"
  ssh-password-option="option.sshPassword1"
  sudo-command-enabled="true"
  sudo-password-storage-path="keys/david"
  file-copier="ssh-copier"
  ssh-options="-o ConnectTimeout=5000"/>
 />
  <node name="server"
  tags="opensuse, linux"
  osFamily="unix"
  username="david"
  hostname="192.168.20.163"
  ssh-authentication="password"
  sudo-command-enabled="true"
  ssh-password-option="option.sshPassword1"
  sudo-command-enabled="true"
  sudo-password-storage-path="keys/david"
  file-copier="ssh-copier"
  ssh-options="-o ConnectTimeout=5000"/>
 />
</project

Now the resources file has been created we need to let Rundeck 3 use it

Edit project properties

vi /var/lib/rundeck/projects/updatelinux/etc/project.properties

Add the following lines to the end of the file

project.execution.history.cleanup.retention.minimum=50 resources.source.1.config.file=/var/lib/rundeck/projects/PROJECTNAME/etc/resources.xml
resources.source.1.config.generateFileAutomatically=true
resources.source.1.config.includeServerNode=true
resources.source.1.type=file

Note: Make sure PROJECTNAME has the actual project name in it.

Save and exit the file

Restart Rundeck

systemctl restart rundeckd

Give it a few minutes to restart, its slow.

Log back in to the Rundeck WebGui

Open the project and select the Nodes option

Make sure all nodes is displayed as a search term and you should see the nodes you entered

Option 2: Using an Import File

This method worked best for me when I need to run multiple projects across the same group of machines.

Create a location Resource

By default there isn't a location for doing this, so we need to add it

mkdir /var/lib/rundeck/resources
chown rundeck:rundeck /var/lib/rundeck/resources

Create the resource File

vi /var/lib/rundeck/resources/resources.xml

add using the following example

<?xml version="1.0" encoding="UTF-8"?>
<project>
<node name="homerepo"
  tags="opensusei, linux"
  osFamily="unix"
  username="david"
  hostname="192.168.86.167"
  ssh-authentication="password"
  sudo-command-enabled="true"
  ssh-password-option="option.sshPassword1"
  sudo-command-enabled="true"
  sudo-password-storage-path="keys/david"
  file-copier="ssh-copier"
  ssh-options="-o ConnectTimeout=5000"/>
 />
</project

if you want to add multiple nodes it would look like this

<?xml version="1.0" encoding="UTF-8"?>
<project>
<node name="proxy"
  tags="opensuse, linux"
  osFamily="unix"
  username="david"
  hostname="192.168.20.167"
  ssh-authentication="password"
  sudo-command-enabled="true"
  ssh-password-option="option.sshPassword1"
  sudo-command-enabled="true"
  sudo-password-storage-path="keys/david"
  file-copier="ssh-copier"
  ssh-options="-o ConnectTimeout=5000"/>
 />
  <node name="server"
  tags="opensuse, linux"
  osFamily="unix"
  username="david"
  hostname="192.168.20.163"
  ssh-authentication="password"
  sudo-command-enabled="true"
  ssh-password-option="option.sshPassword1"
  sudo-command-enabled="true"
  sudo-password-storage-path="keys/david"
  file-copier="ssh-copier"
  ssh-options="-o ConnectTimeout=5000"/>
 />
</project

Now the resources file has been created we can use the Rundeck WebGUI to use it in any project

Open the project

Click on Edit Nodes

In the resulting screen

Click on Add new Node Source

From the resulting menu click on File

Enter the name of the file: resourse.xml

Enter the path: /var/lib/rundeck/resources/

The tick box options are up toy u

Click on Save

Open the project and select the Nodes option

Make sure all nodes is displayed as a search term and you should see the nodes you entered

Update the Default Node Executor

This one took a while to figure out, i was able to get the nodes imported as expected, however when i tried to run one of the tests below, I kept being told that the SSH Password had not been set, this was baffling me because my resource.xml file was pointing to keys/david for the location of the password I setup earlier.

Turns out I needed to update the Default Node Executore (in this case SSH) to also know where the password i'd set was stored.

Open the project the Nodes are connected to and Open Project Settings -> Edit Configuration

Select Default Node Executor at the top

In the SSH Password Storage Path Enter the storage location of your credentials keys/david in my case

Save this

Using Private Key authentication

Using a a keypair is preferred to a password and while the setup for password authentication of nodes takes the same method as the notes I've provided for above there are some notable changes and important learnings to get this working.

I'm basing these instructions on the assumption that you have passwordless SSH setup already between servers and clients. If not you'll need to google Passwordless SSH setup

Convert your private key from SSH to RSA

Your private key will probably be in the format for openssh, Rundeck needs this to be converted to the RSA format

If you show the contents of your private key using

cat /path/to/openssh/key

and the start and end of the key have the following

-----BEGIN OPENSSH PRIVATE KEY-----


-----END OPENSSH PRIVATE KEY-----

Then your private key is in the wrong format for Rundeck and you need to run

ssh-keygen -p -m PEM -f /path/to/openssh/key

This will rewite the existing private key to the RSA format

Viewing the key will see

-----BEGIN RSA PRIVATE KEY-----



-----END RSA PRIVATE KEY-----

You now have a Private key with the right format, if you don't do this the nod will fail with a privateKey format error whey you run the tests below.

Key Storage

Copy and past the new RSA Private key into the Cog -> Key Storage -> Add new Key Window (full Key storage instructions above)

resources.xml

The resources.xlm nodes section will change to look like this

<node name="ubuntu2018"
  tags="ubuntu, linux, server, proxmox"
  osFamily="unix"
  username="david"
  hostname="192.168.20.171"
  ssh-key-storage-path="keys/david"
  sudo-command-enabled="true"
  sudo-password-storage-path="keys/david"
  file-copier="ssh-copier"
  ssh-authentication="privateKey"
  ssh-options="-o ConnectTimeout=5000"/>
 />

The change specifically the lines

ssh-key-storage-path="keys/david"
ssh-authentication="privateKey"

Node Executor

Edit the project config

Under the Default Node Executor Section

Set the config as per the previous instructions except at this point set SSH Authentication to privateKey

Save

Testing Nodes are working.

Once the nodes are imported, you want to know that they are working, and to do this, we can run a simple command line job on them from the Rundeck Web Gui

Open the Project and click on the Nodes tab

Under Actions click on Run Action on X nodes

Within this screen we can run a command on the remote node(s) which in this case i'm going to run the ls command

Click on the green Run on x Nodes button

If all the connectivity above has been set right, you will see the resulting output on the webgui

Notes

Now we have the core project and nodes setup for a project, what we can see is how under the project directory

/var/lib/rundeck/projects/<PROJECTNAME>

we have a folder structure which becomes git commitable, and we can backup

Within the project folders etc directory take a look at

more project.properties

and between this file and resources.xml we can see how the project is defined as config code

project.jobs.gui.groupExpandLevel=1
resources.source.1.config.generateFileAutomatically=true
project.ssh-authentication=password
resources.source.1.config.description=OrangeMo
service.FileCopier.default.provider=jsch-scp
project.execution.history.cleanup.retention.days=60
project.execution.history.cleanup.enabled=false
project.ssh-password-storage-path=keys/david
project.execution.history.cleanup.schedule=0 0 0 1/1 * ? *
project.output.allowUnsanitized=false
project.execution.history.cleanup.batch=500
project.ssh-command-timeout=0
project.disable.executions=false
project.ssh-keypath=/var/lib/rundeck/.ssh/id_rsa
project.description=Update Linux Servers
resources.source.1.config.includeServerNode=true
service.NodeExecutor.default.provider=jsch-ssh
project.name=updatelinux
project.disable.schedule=false 
project.ssh-connect-timeout=0
resources.source.1.type=file
project.label=update
resources.source.1.config.file=/var/lib/rundeck/projects/updatelinux/etc/resources.xml
project.execution.history.cleanup.retention.minimum=50

Blog Version

Links

RunDeck Series 2 - Add Nodes to the Rundeck

Add nodes to rundeck 3.0+ using resources.xml. Also we can import exsiting project nodes to another project. We have 3 ways to add nodes.

BhuvaneshThe Data Guy