Back in June 2020 I wrote a post going over how to install AWX on Ubuntu 18.04. It turns out the instructions quite usefully work on Ubuntu 20.04 as well.

Following many months of successfully running Rundeck to complete the background automation tasks on my home network using mainly Bash scripts I had a catastrophic disk failier on the Gigabrix which was running Rundeck.

Having replaced the disk, I thought i'd set myself a learning challenge of repeating the setup using AWX and Ansible. I find learning far easier if I have a thing to do around the subject as opposed to following a youtube tutorial. Once i've  done a few hours and got my head around a subject those YouTube tutorials make much more sense as I know the questions I need to ask/answer.

So this week I've setup an AWX server at home, a local Gitlab server and a server to run Ansible from and set about working out how to do what I was doing as bash scripts in Ansible.

This post is my method of getting from having never used AWX to getting jobs running on a schedule with notifications that are managed through SCM/Gitlab

Disclaimer

Pick holes, crack on, its what trolls and the internet does. I get that.
I know my grammar is terrible, my spelling sucks, i've probably got it "all wrong". IF your feedback is constructive and conversational where I can learn and share we will get on like a house on fire. If you're going to tell me how I missed a comma or spelt something wrong.. Jog on, i've no interest.*

*This is my journey.. *

What is AWX?

AWX is a centralised server with a web gui for running Ansible jobs pulled from various SCM locations onto hosts grouped as Inventories.

In the Red Hat world AWX is the community driven source for the commerial varian Ansible Tower. AWX is Fedora to Ansibile Tower being Red Hat Enterprise Linux.

Installing

I covered the installation of AWX as a Single box here. I'm not installing a fully fault tolerant, failover, multi server solution. I've got my MariaDb box on a seperate server, other than that AWX is all running on a single server.

Installing AWX on Ubuntu 18.04
The following Tutorial will take you through installing AWX the communityversion of Ansible Tower on Ubuntu 18.04. In this example the Docker Composeversion of AWX will be used > There are a few gotchas picked up from installing this that most howto guidesmissMinimum Spec Machine * Ubuntu 20.0…

Setup

Its probably worth covering quickly what I'm running here. The Machine is an i5 with a 1Tb SSD and 8Gb Ram. (Its a little Gigabrix box) Its running Ubuntu 20.04

On the OS i've got AWX, Documize and Gitlab-ce installed

I have an Ubuntu 20.04 server running in Proxmox which i'm using as a host to run the AWX Ansible jobs  (hostserver). This has access to all the servers on the lan.

I'm running jobs which backup this blog, update lets encrypt certificates, move media files around, backup servers like Gitlab to external storage, update docker images. Nothing too taxing and the code for all of these is held in the Gitlab on the same box as AWX.

Dashboard

When you first login to AWX i can't tell you're dashboard will look quite this interesting. This is where you start your AWX journey

While dashboards give a nice overview of what is happening on the server while we are setting up jobs to run from the box its the resources area of the menu to the left which is the first port of call.

Credentials

The first step is to setup credentials for connecting to the HostServer Ansible box the scripts will be run from

Click on Credentials, on a fresh install there will be a Demo account setup

To add a new credential to the server click on the + symbol in the top right.

This will present you with a screen where you can fill in the Name and Description (I've not set up organizations)

The type of credential is found by clicking on the magnifying glass under Credential Type which will bring up a long list of predefined credential types

As we are connecting to a host server we need to search and find for the Credential type Machine.

Click on Select and you will see that the options for the Machine type are added

Either password only, certificate or certificate and password can be setup here.

In my setup i've added the SSH Private Key, Privilage Esculation Username and Privilage Esculation Password and because i'm lazy i've used the same user and key combination for most of my machines on this project.

Once you've setup the method for connecting to the server thats going to run Ansible click on Save

You will come back to this screen several times to add credentials for other systems, on my setup i've added creds for my local Gitlab setup and Ansible Galaxy

Inventory

Once the credentials have been setup to access remote systems, the next step is to add the inventory items to AWX. These are the remote servers (like my host server) you'll be asking AWX to connect to to run Ansible.

New Inventories are added as with credentials by clicking on the + symbol in the to right.

Note:
Its worth noting that an inventory can contain multiple hosts. Hosts are what you are connecting to.

Understanding this I've setup an Inventory called Homelan and the fields which need filling in are Name and Description

Click on Hosts at the top

Add a new host by clicking on the green + icon

You'll need to add a Host name and Description which are free form fields

Under Variables the ansible host command points to either the IP or DNS name of the host.

---

ansible_host: 192.168.23.250

Click on Save and repeat for each host on your home lan you want to add.

Inventory Testing

Once the hosts have been added to the inventory item the conectivity needs to be tested

Click on the tick next to the host(s) to test against then click on Run Command

Select Ping from the drop down module and search under Machine Credentials for the creds we setup for the hosts in the previous section.

Select the Credentials

then click on Launch.

The job window will launch and run a ping against the host

Run this against all hosts you add to an Inventory.

Expanding the Inventory

Like the /etc/ansible/host file, inventorys can be created not ust for all machines on a Lan, it can be created for groups of hosts on a Lan. Webservers, Jumpservers, Linux Machines etc. So AWX projects can be targeted against specific inventories.

As an example I have 2 Webservers so I created an Inventory called Webservers with 2 hosts in it, the same hosts are also in the HomeLan Inventory which has all the hosts in it.

Project

Having created credentials to access Machines and Gitlab and provided AWX with the Inventory/Hosts combination to run ansible code on. The next step is to create projects. These are the Ansible code which will be run on the remote hosts.

Note:
I'm very aware there is a lot more oing on that I'm about to cover here, and as i get a better understanding of AWX and Git features I will update this document. This is the minimum i've found to get a project setup on AWX.

On a clean install of AWX there are some demo projects setup, to add a new project click again on the Green + Arrow

AWX supports a wide range of platforms where your Ansible code can be pulled in from. This example is using a locally hosted Gitlab-CE install.

Select Git

This unlocks the data needed to pull down the git code

This is an example of the project I use to backup this blog site. (More on that later)

Some things worth noting here. I'm using the IP of the git server because my home DNS can be a bit flakey. The hostname can be used (and probably should be). The SCM URL is the SSH URL from Gitlab and the SCM credential is the gitlab ssh cred with the private key I setup in the credentials section above.

I've ticked the clean, delete on update and update revision on launch to make sure i'm getting the latest code each time this is run.

I'm going to setup the Gitlab repos to do branch, versioning as I move forward so i can use the version number in the SCM Branch/Tag/Commit section. As it stands as a basic use of a project to get running, this setup works fine in my environment.

Click on Save

Template

Having setup the Credentials, Inventory and Projects the three of these need to be brought together as something AWX can run. This is done in the Template section of AWX

Opening the Templates section will again show some Demo templates

Click on the Green + button again, and select Job Template

This will open a new job template

Free flow fields Name and Description ned to be filled in.

Job Type is either Run or check providing the option to either run the job on the inventory or test what would happen if the job was run, but don't actually run it.

Click on the Search to choose which Inventory (group of Hosts) you want to run the job on.

Click on the Search to choose the Project to be run on the Inventory machines

When the Project is chosen the Playbook dropdown will list all the Yaml files you could run from the project

Select the Credentials you'd use to run against the hosts in Inventory

Under Options select Enable Privilege Esculation (if your ansible will need it, remember we setup SUDO information when we setup the credentials)

For the purpose of this blog, this is the minimum information needed to run a template on AWX. Other options, you can discover on your own.

It is a this point fine to click on Save and launch the job manually and to test it i'd suggest this will be done many times. Once your job is running on the AWX Inventory Hosts the sext step is to add Notifications to tell when the project has started, run or failed and scheduls to say when to run the project

Notifications

(Email URL)

Notifications do just as they say and let you know if a job has started, finished or failed.

Under the Job Template clickon the notification tab. On a brand new install you will be prompted to setup a new notification location to use this option

If you want to setup new notification sources then click on the left hand menu option.

This takes you to Notification Templates

To add a new notification destination Click on the green + symbol to open the New Notification Template window.

AWX can communicate with a good selection of services including email, slack, pagerduty etc.

Select email from the list

the fields Name, Description are free flow fields and need to be completed

The details of your mail server need to be added, the example here I have an internal mail relay for the services on my hme lab which doesn't see the outside world. As such i've not setup any username or password and left it on Port 25

Add who is sending the email and who it should go to

You can add security settings such as TLS using the drop down email options menu.

Click Save

You can test the notifications as well by clicking on the Bell.

Note:

When the notifications are sent through,they arrive with a link to AWX of https://ansible.tower to change this of Settings -> System and change the Base URL of the Tower Host option then save.

Scheduling

Setting up schedulling is pretty quick, open the job template you want to schedule and click on the Schedule tab

Click on the green + button to add a new task schedule

The Create Schedule screen is self explainitory git it a name, when you want the scheduled job to start from, the time in hh:mm:ss format to run the job, which time zone and when to repeat it. then click on Save

The following report is displayed.

Basic AWX is now done

At this point a Basic AWX job is now done. Creds are created, Inventory where to run is created, the Project and Ansible code is linked and the Template connecting all of these has been produced to run on a schedule and notify when it runs.

Is there more to AWX than this, at its core, not really it like Rundeck is a tool for running tasks across servers on a schedule. the core difference is AWX is designed to run ansible jobs and rundeck is designed to run ansible and a myriad of other scripting languages.

Out of the box I think AWX is easier to setup if you already use Ansible, and Ansible while sometimes fustrating (really no move module?) isn't difficult even for an aging technical manager like myself to pick up, I think for integration jobs which may require more languages and control. Rundeck is a better choice.

Troubleshooting

While I was setting things up, I have come across a few things its probably worth sharing here. Most of them relate to my lack of Ansible knowledge than AWX issues.

Community Plugins

This was an interesting one. While creating the Ansible code for backing up this blog i wanted to pull all the backup files into an archive .tar.gz or .tgz type thing and I ended up with this code:

- name: Compress www backup into tgz
  community.general.archive:
    path: "{{ remote_dir }}/"
    dest: "{{ local_dir }}/ghost{{ansible_date_time.date}}.tgz"

When I ran this on my test environment I was informed i'd spelt something wrong in community.general.archive: which translated into I didn't have the Galaxy modules community.general installed.

Identity added: /tmp/awx_68_v4vm0gel/artifacts/68/ssh_key_data (david@Pad430s)
BECOME password:
ERROR! couldn't resolve module/action 'community.general.archive'. This often indicates a misspelling, missing collection, or incorrect module path.The error appears to be in '/tmp/awx_68_v4vm0gel/project/ghost_backup.yaml': line 52, column 7, but may be elsewhere in the file depending on the exact syntax problem.The offending line appears to be: 
- name: Compress www backup into tgz
 ^ here

This was easily rectified and I ran the following on the test box and the hostserver box which was going to run the job from AWX

ansible-galaxy collection install community.general

Running the ansible-playbook command on both boxes solved the above error.

I then ran through setting the job up in AWX and on the first run I again got the message above..

How could this be? The host the template is running on has the ansible-galaxy community.general installed and I tested it.

I knocked up some very hacky ansible to run from AWX to make sure I was running this where I thought i was and who I thought I was

- hosts: 127.0.0.1
  become: true
  vars:

  tasks:
- debug: var=inventory_hostname
- debug: var=ansible_hostname
  #- debug: var=hostvars

- name: get the username running the deploy
  local_action: command whoami
  register: username_on_the_host

- debug: var=username_on_the_host

(Formatting is off, don't copy and paste and expect to run)

This came back confirmed I was running on the expected host and as the expected and same user as if I run the job locally using the ansible-playbook command.

The answer was finally found (for me) here

Hands on with Ansible collections
In this blog post we’ll walk through a use case wherein, the user would like to use a Red Hat certified collection from Automation Hub and also use a community supported collection from Ansible Galaxy.

I think this is my lack of proper Ansible knowledge however it seems I need to have the collection and an ansible.cfg file like such included as part of my Ansible code

[defaults]
stdout_callback = yaml
collections_paths = ./collections

[galaxy]

server_list = release_galaxy

[galaxy_server.release_galaxy]
url=https://galaxy.ansible.com/
token=c1a987349889jfjf9036285e46bfkdk78364389d61

Under collections/requirements.yml this was added

collections:
name: community.generalsource: https://galaxy.ansible.com

then from the root folder of the Ansible code (the one with the yaml and the ansible.cfg i just created in) I ran

ansible-galaxy collection install -r collections/requirements.yml

Adding all of this and pushing to my GitLab repo resolved that issue for me.

Where and I running as who?

Its worth I think having a trouble shooting test script for AWX  as i can see what works using ansible-playbook on the command line may not work in AWX as they are looking in different places

The start of my ansible troubleshooting YAML is this:

- hosts: all
  become: true
  vars:

  tasks:
- debug: var=inventory_hostname
- debug: var=ansible_hostname
  #- debug: var=hostvars

- name: get the username running the deploy
  local_action: command whoami
  register: username_on_the_host

- debug: var=username_on_the_host

It tells me who is running the ansible code and where its being run.

Ansible YAML Files

This is an obvious one once you realise why, and a noob mistake

at the top of the ansible yaml code don't use this or derivitives of

hosts: 127.0.0.1

Use

hosts: all

As AWX will pick up where to run the job and what all is from the inventory that has been set in the template.

Thoughts

As I've reittereated many times here, this is a learning experience and a useful one. While long term Ansible and AWX/Tower users may giggle at my writings, its important to take a step back and realise as Automation becomes more common place, its the tools sysadmins (who are not developers, thats why they are sysadmins) can pick up quickly and get on board with.

AWX at least in this simple home situation where i'm running 20 jobs on different schedules to automate some tasks has been quick and easy to setup. the Google support has been as with most things just learnig the right phrases and there is no shortage of free Ansible advice, examples and tutorials out there for any level.