Introduction
Rundeck is both a hugely powerful method of centralising your scripts and time based (cron) jobs and a right royal pain in the bum to setup. Being an open source project its changed over may iterations and in doing so its an example of finding information on the web when problems arise and because its an older version the resolutions are not easy to come by.
As a project it also needs (for my sysadmin rather than developer mind) you to think a little different from similar tools like Jenkins
When setup however, its a hugely powerful project and well worth investigating
Disclaimer
These notes are best used to get your feet wet with Rundeck, i'm using password based SSH authentication in the initial release of this post (which will get updated over time) and id rather be using Keys, i'm not using the database back end and i'm learning as I write this. I'm happy to take measured feedback and update/credit the best place to do this is via reddit.
https://www.reddit.com/user/mightywomble
Environment
I'm running this in the following environment
CentOS 7
4Gb Ram
250Gb HDD
All commands run as root (via sudo)
Install
Setup OS
Update the OS
yum update
Disable SELinux
vi /etc/selinux/config
Change
SELINUX=enabled
to
SELINUX=disabled
Reboot your system
reboot
Install Rundeck
Install the latest repo data
rpm -Uvh http://repo.rundeck.org/latest.rpm
Then install the packages
sudo yum install rundeck java rundeck-cli
Open Ports
The Centos Firewall blocks all to start with, you will need to open port 4440/tcp
firewall-cmd --add-port=4440/tcp --permanent
firewall-cmd --reload
Edit Settings
The rundeck config files need editing so when you click on links in the Rundeck Gui they work properly remotely
vi /etc/rundeck/rundeck-config.properties
Change the lines
grails.serverURL=http://localhost:4440
to
grails.serverURL=http://rundeck:4440
then edit the framework file
vi /etc/rundeck/framework.properties
change
framework.server.name = localhost
framework.server.hostname = localhost
framework.server.port = 4440
framework.server.url = http://localhost:4440
to
framework.server.name = rundeck
framework.server.hostname = rundeck
framework.server.port = 4440
framework.server.url = http://rundeck:4440
Change the Rundeck Login Password
The Default login/password for Rundeck is admin/admin and we should change this.
vi /etc/rundeck/realm.properties
change the line
admin:admin,user,admin,architect,deploy,build
to
admin:YoUrNeWpaSsWoRdHeRe,user,admin,architect,deploy,build
save and exit
Create Projects in the file system not the Database
Note: There are a lot of online instructions which point to the working directory on the file system, however on the version I installed projects data is saved in a Db, while i'm learning the system and little documentation on the Rundeck site at the time of writing has details about the Db side I've changed the install config to create projects in the file system
The location the projects are stored in is /var/lib/rundeck/projects
vi /etc/rundeck/rundeck-config.properties
find the line
rundeck.projectsStorageType=db
change it to
rundeck.projectsStorageType=file
save and exit
Start Rundeck
Enable Rundeck so it starts after a reboot
systemctl enable rundeckd
Start the service
systemctl start rundeckd
Rundeck takes a few minutes to start, and we can check the service log file for indication
tail -f /var/log/rundeck/service.log
If the following line is seen
Grails application running at http://localhost:4440 in environment: production
The server has started
Login
Open Rundeck in the browser
http://rundeck:4440
You cal login using the admin and password you set

This will open the rundeck gui

Setup Project
The core of Rundecks world is projects, projects are the overarching containers for nodes and jobs, you can have multiple nodes or jobs in a Project
Access control to projects can be applied as well, allowing RBAC linked to LDAP/AD to resent users with projects they should have access to.
Create a new project, and give it a name, label and description

Click on Create
Once created the project view in the WebGUI provides a more functional interface

Setup Remote Nodes
Setting up nodes was the first stumbling block I had on this journey as the setup points project config to the DB by default. During the setup above I had the projects be managed as files rather than a database.
Having had projects create as a files and realising /var/lib/rundeck was the root application folder not /var/rundeck as many instructions state. I've found two methods of setting up nodes
SECURITY NOTE: I am aware that ideally this should be done using passwordless ssh keys, and once my setup is working i'll add this option.
However before we use ither of them we need to setup our password in the Rundesck Key Storage
Open the Key Storage page under the Cog to the top right

Select Key Storage

Click on Add Key

Select password from the dropdown
In Enter Text add your password
In name enter the username
Click on Save

Make a note of the storage path
In this example keys/david
Options for adding Nodes
Option 1: Per project using the command line
This option works best if you need different groups of nodes per project
Navigate to your project folder on the Rundeck server
cd /var/lib/rundeck/projects/PROJECTNAME/etc
create a new file resources.xml
vi resources.xml
add using the following example
<?xml version="1.0" encoding="UTF-8"?>
<project>
<node name="homerepo"
tags="opensusei, linux"
osFamily="unix"
username="david"
hostname="192.168.86.167"
ssh-authentication="password"
sudo-command-enabled="true"
ssh-password-option="option.sshPassword1"
sudo-command-enabled="true"
sudo-password-storage-path="keys/david"
file-copier="ssh-copier"
ssh-options="-o ConnectTimeout=5000"/>
/>
</project
if you want to add multiple nodes it would look like this
<?xml version="1.0" encoding="UTF-8"?>
<project>
<node name="proxy"
tags="opensuse, linux"
osFamily="unix"
username="david"
hostname="192.168.20.167"
ssh-authentication="password"
sudo-command-enabled="true"
ssh-password-option="option.sshPassword1"
sudo-command-enabled="true"
sudo-password-storage-path="keys/david"
file-copier="ssh-copier"
ssh-options="-o ConnectTimeout=5000"/>
/>
<node name="server"
tags="opensuse, linux"
osFamily="unix"
username="david"
hostname="192.168.20.163"
ssh-authentication="password"
sudo-command-enabled="true"
ssh-password-option="option.sshPassword1"
sudo-command-enabled="true"
sudo-password-storage-path="keys/david"
file-copier="ssh-copier"
ssh-options="-o ConnectTimeout=5000"/>
/>
</project
Now the resources file has been created we need to let Rundeck 3 use it
Edit project properties
vi /var/lib/rundeck/projects/updatelinux/etc/project.properties
Add the following lines to the end of the file
project.execution.history.cleanup.retention.minimum=50 resources.source.1.config.file=/var/lib/rundeck/projects/PROJECTNAME/etc/resources.xml
resources.source.1.config.generateFileAutomatically=true
resources.source.1.config.includeServerNode=true
resources.source.1.type=file
Note: Make sure PROJECTNAME has the actual project name in it.
Save and exit the file
Restart Rundeck
systemctl restart rundeckd
Give it a few minutes to restart, its slow.
Log back in to the Rundeck WebGui
Open the project and select the Nodes option

Make sure all nodes is displayed as a search term and you should see the nodes you entered
Option 2: Using an Import File
This method worked best for me when I need to run multiple projects across the same group of machines.
Create a location Resource
By default there isn't a location for doing this, so we need to add it
mkdir /var/lib/rundeck/resources
chown rundeck:rundeck /var/lib/rundeck/resources
Create the resource File
vi /var/lib/rundeck/resources/resources.xml
add using the following example
<?xml version="1.0" encoding="UTF-8"?>
<project>
<node name="homerepo"
tags="opensusei, linux"
osFamily="unix"
username="david"
hostname="192.168.86.167"
ssh-authentication="password"
sudo-command-enabled="true"
ssh-password-option="option.sshPassword1"
sudo-command-enabled="true"
sudo-password-storage-path="keys/david"
file-copier="ssh-copier"
ssh-options="-o ConnectTimeout=5000"/>
/>
</project
if you want to add multiple nodes it would look like this
<?xml version="1.0" encoding="UTF-8"?>
<project>
<node name="proxy"
tags="opensuse, linux"
osFamily="unix"
username="david"
hostname="192.168.20.167"
ssh-authentication="password"
sudo-command-enabled="true"
ssh-password-option="option.sshPassword1"
sudo-command-enabled="true"
sudo-password-storage-path="keys/david"
file-copier="ssh-copier"
ssh-options="-o ConnectTimeout=5000"/>
/>
<node name="server"
tags="opensuse, linux"
osFamily="unix"
username="david"
hostname="192.168.20.163"
ssh-authentication="password"
sudo-command-enabled="true"
ssh-password-option="option.sshPassword1"
sudo-command-enabled="true"
sudo-password-storage-path="keys/david"
file-copier="ssh-copier"
ssh-options="-o ConnectTimeout=5000"/>
/>
</project
Now the resources file has been created we can use the Rundeck WebGUI to use it in any project
Open the project

Click on Edit Nodes
In the resulting screen

Click on Add new Node Source

From the resulting menu click on File

Enter the name of the file: resourse.xml
Enter the path: /var/lib/rundeck/resources/
The tick box options are up toy u
Click on Save

Open the project and select the Nodes option

Make sure all nodes is displayed as a search term and you should see the nodes you entered
Update the Default Node Executor
This one took a while to figure out, i was able to get the nodes imported as expected, however when i tried to run one of the tests below, I kept being told that the SSH Password had not been set, this was baffling me because my resource.xml file was pointing to keys/david for the location of the password I setup earlier.
Turns out I needed to update the Default Node Executore (in this case SSH) to also know where the password i'd set was stored.
Open the project the Nodes are connected to and Open Project Settings -> Edit Configuration
Select Default Node Executor at the top

In the SSH Password Storage Path Enter the storage location of your credentials keys/david in my case

Save this
Using Private Key authentication
Using a a keypair is preferred to a password and while the setup for password authentication of nodes takes the same method as the notes I've provided for above there are some notable changes and important learnings to get this working.
I'm basing these instructions on the assumption that you have passwordless SSH setup already between servers and clients. If not you'll need to google Passwordless SSH setup
Convert your private key from SSH to RSA
Your private key will probably be in the format for openssh, Rundeck needs this to be converted to the RSA format
If you show the contents of your private key using
cat /path/to/openssh/key
and the start and end of the key have the following
-----BEGIN OPENSSH PRIVATE KEY-----
-----END OPENSSH PRIVATE KEY-----
Then your private key is in the wrong format for Rundeck and you need to run
ssh-keygen -p -m PEM -f /path/to/openssh/key
This will rewite the existing private key to the RSA format
Viewing the key will see
-----BEGIN RSA PRIVATE KEY-----
-----END RSA PRIVATE KEY-----
You now have a Private key with the right format, if you don't do this the nod will fail with a privateKey format error whey you run the tests below.
Key Storage
Copy and past the new RSA Private key into the Cog -> Key Storage -> Add new Key Window (full Key storage instructions above)

resources.xml
The resources.xlm nodes section will change to look like this
<node name="ubuntu2018"
tags="ubuntu, linux, server, proxmox"
osFamily="unix"
username="david"
hostname="192.168.20.171"
ssh-key-storage-path="keys/david"
sudo-command-enabled="true"
sudo-password-storage-path="keys/david"
file-copier="ssh-copier"
ssh-authentication="privateKey"
ssh-options="-o ConnectTimeout=5000"/>
/>
The change specifically the lines
ssh-key-storage-path="keys/david"
ssh-authentication="privateKey"
Node Executor
Edit the project config

Under the Default Node Executor Section

Set the config as per the previous instructions except at this point set SSH Authentication to privateKey
Save
Testing Nodes are working.
Once the nodes are imported, you want to know that they are working, and to do this, we can run a simple command line job on them from the Rundeck Web Gui
Open the Project and click on the Nodes tab
Under Actions click on Run Action on X nodes

Within this screen we can run a command on the remote node(s) which in this case i'm going to run the ls command

Click on the green Run on x Nodes button

If all the connectivity above has been set right, you will see the resulting output on the webgui
Notes
Now we have the core project and nodes setup for a project, what we can see is how under the project directory
/var/lib/rundeck/projects/<PROJECTNAME>
we have a folder structure which becomes git commitable, and we can backup
Within the project folders etc directory take a look at
more project.properties
and between this file and resources.xml we can see how the project is defined as config code
project.jobs.gui.groupExpandLevel=1
resources.source.1.config.generateFileAutomatically=true
project.ssh-authentication=password
resources.source.1.config.description=OrangeMo
service.FileCopier.default.provider=jsch-scp
project.execution.history.cleanup.retention.days=60
project.execution.history.cleanup.enabled=false
project.ssh-password-storage-path=keys/david
project.execution.history.cleanup.schedule=0 0 0 1/1 * ? *
project.output.allowUnsanitized=false
project.execution.history.cleanup.batch=500
project.ssh-command-timeout=0
project.disable.executions=false
project.ssh-keypath=/var/lib/rundeck/.ssh/id_rsa
project.description=Update Linux Servers
resources.source.1.config.includeServerNode=true
service.NodeExecutor.default.provider=jsch-ssh
project.name=updatelinux
project.disable.schedule=false
project.ssh-connect-timeout=0
resources.source.1.type=file
project.label=update
resources.source.1.config.file=/var/lib/rundeck/projects/updatelinux/etc/resources.xml
project.execution.history.cleanup.retention.minimum=50
Blog Version
Date | Version Number | Description | Notes | References |
16/05/2020 | 1.0.0 | Initial Blog Post | None | None |
19/05/2020 | 1.1.0 | Added Instuctions to use Private Key in Nodes | ||