| Version | Date | Description |
| 1.0 | 11 May 2021 | Initial Post |
| 1.2 | 20 May 2021 | the code in the GitLab link at the end of the post is now updated to deploy using HTTPS and use the API over HTTPS. |
While working on a personal project and getting to a version 1 release, there was an obvious problem with my code (other than it being written by a sysadmin).
There were passwords in the playbooks which even I know is not a good thing.
It's not good for security, however, it's also not good for portability or immutability.
The question was how best to store these passwords?
The obvious answer is Ansible Vault however because of the end goal of the project I was looking for something with possibly a WebUI and obviously something which integrated well with Ansible.
Hashicorp Vault was mentioned and so the journey began from having never used Hashicorp Vault to installing it sing Ansible and using it to store passwords and pull them back out.
End Result
The art of moving forward is knowing where you want to get to.. my destination was a simple one. Automate using ansible the following:
- deploys HCP Vault
- unseals HCP Vault
- creates a new secret repository
- populates the new repository with random secrets for the services which need them
- has a role to pull the secret out for the service and use it in a playbook.
Essentially going from zero to extracting as-needed service-related random passwords.
Working environment
I have Ansible AWX setup on my home lab and all code is pushed to gitlab, and then run on the remote RHEL8 Dev server from AWX.
This will mean some of the code might need "tweaking" if you just want to run "ansible-playbook" to run this.
Why write this?
Turns out using google to get the output I needed lead me down a lot of rabbit holes to end up with something which worked for me. Hopefully, if you find this it might save you a few Rabbit holes and mistyped Stack Overflow pages.
It's also notable that many of the pages are written from a developers point of view, something I'm not, so I've written this very much from the point of view of a sysadmin.
Disclaimer
Before going ahead with this, let's deal with the trolls and keyboard warriors who love a bit of a whine because it doesn't fit their narrow view of the world.
Not Production Code
I cannot highlight this enough, while this code works, it "works for me" and "works on my dev environment" The point of this post is not to promote this a finished solution. It is a guide to things that may or may not work for you.
become_root
Yes, I know, it's for development ease of use..
Debugging Enabled
no_log isn't enabled and the code has several debug statements in it which will display the passwords in the AWX output.
No Guru
I'm learning this, I'm not a code guru, I'm someone who likes a challenge, will use google to find the answers and adapt accordingly to understand what I'm writing.
I'm open to constructive, explained feedback.
Get Started
The file structure is important as Ansible and AWX will not be able to find the files it needs if its in the wrong place
service_deployment
|-> collections
|-> playbook
| -> files
|- **services**
| -> roles
| -> vaultpwd
| -> tasks
| - **main.yaml**
| -> library
| -> templates
| - **config.hcl.j2**
|- **prerequsw.yaml**
|- **install_vault.yaml**
|- **add_vault_passwords.yaml**
|- **install_mariadb.yaml**
|-> **onehit.yaml**
In the root of the Ansible play there is the file I'm using to run this all.
onehit.yaml
---
- hosts: all
become: root
- name: (One Hit) - Pre Req
import_playbook: playbook/prerequsw.yaml
- name: (One Hit) - Install Vault
import_playbook: playbook/install_vault.yaml
- name: (One Hit) - Add Services
import_playbook: playbook/add_vault_passwords.yaml
- name: (One Hit) - Testing Role with Dashboard
import_playbook: playbook/install_mariadb.yaml
Under playbook there are 4 playbooks
prerequsw.yaml
- hosts: all
become_user: root
tasks:
- name: install PIP3
yum:
name:
- python3
- python3-pip
state: latest
install_vault.yaml
- hosts: all
become_user: root
environment:
VAULT_ADDR: http://127.0.0.1:8200
vars:
vaultdata: /opt/vault/data
vaultip: "0.0.0.0:8200"
unseal_keys_dir_output: /root/vault/keys #fordebug
root_token_dir_output: /root/vault/tokens #fordebug
tasks:
##Install Prereqs
- name: (VAULT - Pre Reqs) install dependencies
yum:
name: yum-utils
state: latest
- name: (VAULT - Pre Reqs) pip install ply version 3.8 for hvac[parser]
pip:
name: ply
version: "3.8"
extra_args: --user
- name: (VAULT - Pre Reqs) Install python packages
pip:
name:
- hvac
- hvac[parser]
extra_args: --user
##Install Hashicorp Vault
- name: (VAULT - Install) Add repository
yum_repository:
name: hashicorp
description: Hashicorp Repository
baseurl: https://rpm.releases.hashicorp.com/RHEL/$releasever/$basearch/stable
gpgkey: https://rpm.releases.hashicorp.com/gpg
- name: (VAULT - Install) install Hashicorp Vault
yum:
name: vault
state: latest
- name: (VAULT - Install) Create a directory if it does not exist
ansible.builtin.file:
path:
- "{{ vaultdata }}"
state: directory
recurse: yes
owner: vault
group: vault
mode: '0777'
- name: (VAULT - Install) Remove file vault.hcl (delete file)
ansible.builtin.file:
path: /etc/vault.d/vault.hcl
state: absent
- name: (VAULT - Install) Copy Config file
template:
dest: /etc/vault.d/vault.hcl
src: config.hcl.j2
owner: vault
group: vault
mode: '0644'
- name: (VAULT - Install) Start vault service
systemd:
state: restarted
name: vault
enabled: yes
daemon_reload: yes
- name: (VAULT - Install) Open port 8200
ansible.posix.firewalld:
port: 8200/tcp
permanent: yes
state: enabled
- name: (VAULT - Install) reload service firewalld
systemd:
name: firewalld
state: reloaded
## Create Hashicorp Vault keys and token
- name: (VAULT - Keys) Create unseal directories
file:
path: "{{ unseal_keys_dir_output }}"
state: directory
- name: (VAULT - Keys) Create root key directories
file:
path: "{{ root_token_dir_output }}"
state: directory
- name: (VAULT - Keys) Initialise Vault operator
shell: vault operator init -key-shares=5 -key-threshold=3 -format json
environment:
VAULT_ADDR: "http://127.0.0.1:8200"
register: vault_init_results
- name: (VAULT - Keys) Parse output of vault init
set_fact:
vault_init_parsed: "{{ vault_init_results.stdout | from_json }}"
- name: (VAULT - Keys) Write unseal keys to files
copy:
dest: "{{ unseal_keys_dir_output }}/unseal_key_{{ item.0 }}"
content: "{{ item.1 }}"
with_indexed_items: "{{ vault_init_parsed.unseal_keys_hex }}"
- name: (VAULT - Keys) Write root token to file
copy:
content: "{{ vault_init_parsed.root_token }}"
dest: "{{root_token_dir_output}}/rootkey"
- name: (VAULT - Keys) set toot token as fact
set_fact:
vault_token: "{{ vault_init_parsed.root_token }}"
cacheable: yes
- debug: msg="{{ vault_token }}"
- name: (VAULT - Install) Add environmental vars
blockinfile:
path: /etc/environment
block: |
export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN="{{ vault_token }}"
## unseal vault
- name: (VAULT - Unseal) Reading unseal key contents
command: cat {{item}}
register: unseal_keys
with_fileglob: "{{ unseal_keys_dir_output }}/*"
- name: (VAULT - Unseal) Unseal vault with unseal keys
shell: |
vault operator unseal {{ item.stdout }}
environment:
VAULT_ADDR: "http://127.0.0.1:8200"
with_items: "{{unseal_keys.results}}"
add_vaultpasswords.yaml
---
- hosts: all
become_user: root
### References
vars:
vault_url: "http://{{ ansible_host}}:8200"
services: "{{ playbook_dir }}/files/services"
secretstore: homeserver
tasks:
- name: (VAULT - Keys) set toot token as fact
set_fact:
vault_url: "http://{{ ansible_host}}:8200"
cacheable: yes
- debug: msg="{{ vault_url }}"
- name: Wait 2 minutes
pause:
minutes: "2"
- name: (VAULT - Populate) Create Keystore
environment:
VAULT_TOKEN: "{{ vault_token|quote }}"
VAULT_ADDR: "{{ vault_url|quote }}"
shell: vault secrets enable -version=1 -path "{{ secretstore }}" kv
- name: (VAULT - Populate) Add services to Vault
vars:
rndpass: "{{ lookup('password', '/dev/null length=15 chars=ascii_letters,digits') }}"
uri:
url: "{{ vault_url }}/v1/homeserver/{{ item }}"
body_format: json
method: POST
headers:
X-Vault-Token: "{{ vault_token }}"
body: { "options": { "max_versions": "12" }, "data": { "password":"{{ rndpass }}" }}
return_content: yes
status_code: "204"
no_log: True
with_items: "{{ lookup('file', '{{ services }}').splitlines() }}"
install_mariadb.yaml
---
- hosts: all
become_user: root
vars:
#dbrootpass: CHFEqGGirtddBQjoM6LLy9i6CQK
servicename: mariadbroot
roles:
- vaultpwd
tasks:
### Setup MariaDB
- name: DNF - Install MariaDB
dnf:
name:
- mariadb-server
- mariadb
state: present
update_cache: True
- name: FIREWALLD - permit traffic in default zone for mysql service
ansible.posix.firewalld:
service: mysql
permanent: yes
state: enabled
- name: SYSTEMD - Start Mysql Service
service:
name: mariadb
state: started
enabled: true
- name: Creating DB Creds
copy:
dest: "/root/.my.cnf"
content: |
[client]
user=root
password="{{ servicepwd }}"
- import_tasks: mysql-python_dependencies.yml
- name: use "mysql_secure_installation" for Fresh MySQL Installation
mysql_secure_installation:
login_password: ''
new_password: "{{ servicepwd }}"
user: root
login_host: localhost
hosts: ['localhost', '0.0.0.0', '192.168.1.30', '192.168.1.34', '172.17.0.3', '127.0.0.1', '::1']
change_root_password: true
remove_anonymous_user: true
disallow_root_login_remotely: false
remove_test_db: true
register: secure
- debug:
var: secure
Under playbook/templates there are some J2 templates
config.hcl.j2 (referenced in install_vault.yaml)
# Full configuration options can be found at https://www.vaultproject.io/docs/configuration
ui = true
#mlock = true
#disable_mlock = true
storage "raft" {
path = "/opt/vault/data"
node_id = "node1"
}
# HTTP listener
listener "tcp" {
address = "0.0.0.0:8200"
tls_disable = 1
}
# HTTPS listener
#listener "tcp" {
# address = "0.0.0.0:8200"
# tls_cert_file = "/opt/vault/tls/tls.crt"
# tls_key_file = "/opt/vault/tls/tls.key"
#}
api_addr = "http://127.0.0.1:8200"
cluster_addr = "https://127.0.0.1:8201"
Under playbook/roles there is a simple role
vaultpwd/tasks/main.yaml
- name: (VAULT - Extract) Pull pwd data out of vault
uri:
url: "{{ vault_url }}/v1/homeserver/{{ servicename }}"
body_format: json
method: GET
return_content: yes
headers:
X-Vault-Token: "{{ vault_token }}"
register: data
- name: Print returned json dictionary
debug:
var: data.json
- name: Print certain element
debug:
var: data.json.data.data['password']
- name: set_fact some paramater
set_fact:
servicepwd: "{{ data.json.data.data['password'] }}"
- debug: msg="{{ servicepwd }}"
Under files, there is a reference file
playbook/files/services
ldap
dashboard
kasmuser
kasmadmin
mariadbroot
nextclouddb
onlyofficedb
wgadmin
wgadminapi
wireguard_priv
What's going on here?
So this is a lot of files, I'll try and break down my thought process here.
onehit.yaml is the file that kicks all this off and what AWX executes on the remote RHEL8 server.
Install needed packages
The first Playbook run is
- name: (One Hit) - Pre Req
import_playbook: playbook/prerequsw.yaml
Because this is a test branch of the main development there are some prerequisite software missing which needed installing, specifically python.
- name: install PIP3
yum:
name:
- python3
- python3-pip
state: latest
I have use yum to do the install, longer-term I'll migrate this to dnf
I've tried to keep the python modules needed within the playbooks which need them as I find this easier to debug later.
Install Vault
The next onehit.yaml Playbook run is:
- name: (One Hit) - Install Vault
import_playbook: playbook/install_vault.yaml
This installs vault, it uses the version in the RHEL/Centos Repo's as directed on the HCP Vault install page
The code needs tidying up however is broken down into the following sections
First, the required software is installed
- name: (VAULT - Pre Reqs) install dependencies
yum:
name: yum-utils
state: latest
- name: (VAULT - Pre Reqs) pip install ply version 3.8 for hvac[parser]
pip:
name: ply
version: "3.8"
extra_args: --user
- name: (VAULT - Pre Reqs) Install python packages
pip:
name:
- hvac
- hvac[parser]
#extra_args: --user
The Pip Modules are a requirement of the hashi.vault galaxy collection and I've not used this because of a limitation of the AWX version I'm running with lookup functions.
If you're using the hashi.vault collection then the hvac is needed of the lookups will bork out.
Next Vault is installed
- name: (VAULT - Install) Add repository
yum_repository:
name: hashicorp
description: Hashicorp Repository
baseurl: https://rpm.releases.hashicorp.com/RHEL/$releasever/$basearch/stable
gpgkey: https://rpm.releases.hashicorp.com/gpg
- name: (VAULT - Install) install Hashicorp Vault
yum:
name: vault
state: latest
Next, the location of the encrypted data for the vault is created, I'm using locally stored data because of the project this relates to.
The default hcl config file is removed and the j2 template is used to replace it.
It's important here to make sure that the new vault.hcl is owned by vault and the mode is set correctly or vault will not start.
The J2 in this case is just a text file, it needs the variables added for the listener, API and cluster address added.
- name: (VAULT - Install) Create a directory if it does not exist
ansible.builtin.file:
path:
- "{{ vaultdata }}"
state: directory
recurse: yes
owner: vault
group: vault
mode: '0777'
- name: (VAULT - Install) Remove file vault.hcl (delete file)
ansible.builtin.file:
path: /etc/vault.d/vault.hcl
state: absent
- name: (VAULT - Install) Copy Config file
template:
dest: /etc/vault.d/vault.hcl
src: config.hcl.j2
owner: vault
group: vault
mode: '0644'
Finally, the vault is started and the required firewall ports opened locally.
- name: (VAULT - Install) Start vault service
systemd:
state: restarted
name: vault
enabled: yes
daemon_reload: yes
- name: (VAULT - Install) Open port 8200
ansible.posix.firewalld:
port: 8200/tcp
permanent: yes
state: enabled
- name: (VAULT - Install) reload service firewalld
systemd:
name: firewalld
state: reloaded
with Vault installed, it is however sealed, manually to unseal a Vault instance 5 keys and token are created and 3 of the keys can be used to unseal the vault
The command below is issued
vault operator init
Which would create
Unseal Key 1: 4jYbl2CBIv6SpkKj6Hos9iD32k5RfGkLzlosrrq/JgOm
Unseal Key 2: B05G1DRtfYckFV5BbdBvXq0wkK5HFqB9g2jcDmNfTQiS
Unseal Key 3: Arig0N9rN9ezkTRo7qTB7gsIZDaonOcc53EHo83F5chA
Unseal Key 4: 0cZE0C/gEk3YHaKjIWxhyyfs8REhqkRW/CSXTnmTilv+
Unseal Key 5: fYhZOseRgzxmJCmIqUdxEm9C3jB5Q27AowER9w4FC2Ck
Initial Root Token: s.KkNJYWF5g0pomcCLEmDdOVCW
Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.
Vault does not store the generated master key. Without at least 3 key to
reconstruct the master key, Vault will remain permanently sealed!
It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.
while looking at how to do this in Ansible there were a lot (a lot) of failed attempts which I think may be down to the AWX docker setup I'm running.
The resulting code runs the vault operator init code and captures the results as a variable, and then parses the results to perform the unseal operation.
In this example, the keys and tokens are held on the server under /root/vault/
- name: (VAULT - Keys) Create unseal directories
file:
path: "{{ unseal_keys_dir_output }}"
state: directory
#delegate_to: localhost
- name: (VAULT - Keys) Create root key directories
file:
path: "{{ root_token_dir_output }}"
state: directory
- name: (VAULT - Keys) Initialise Vault operator
shell: vault operator init -key-shares=5 -key-threshold=3 -format json
environment:
VAULT_ADDR: "http://127.0.0.1:8200"
register: vault_init_results
- name: (VAULT - Keys) Parse output of vault init
set_fact:
vault_init_parsed: "{{ vault_init_results.stdout | from_json }}"
- name: (VAULT - Keys) Write unseal keys to files
copy:
dest: "{{ unseal_keys_dir_output }}/unseal_key_{{ item.0 }}"
content: "{{ item.1 }}"
with_indexed_items: "{{ vault_init_parsed.unseal_keys_hex }}"
- name: (VAULT - Keys) Write root token to file
copy:
content: "{{ vault_init_parsed.root_token }}"
dest: "{{root_token_dir_output}}/rootkey"
- name: (VAULT - Keys) set toot token as fact
set_fact:
vault_token: "{{ vault_init_parsed.root_token }}"
cacheable: yes
- debug: msg="{{ vault_token }}"
- name: (VAULT - Install) Add environmental vars
blockinfile:
path: /etc/environment
block: |
export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN="{{ vault_token }}"
With the keys separated into their own files, the following ansible code will unlock the vault. the command vault operator unseal then runs over a loop against the vault server.
- name: (VAULT - Unseal) Reading unseal key contents
command: cat {{item}}
register: unseal_keys
with_fileglob: "{{ unseal_keys_dir_output }}/*"
- name: (VAULT - Unseal) Unseal vault with unseal keys
shell: |
vault operator unseal {{ item.stdout }}
environment:
VAULT_ADDR: "http://127.0.0.1:8200"
with_items: "{{unseal_keys.results}}"
This will unlock the vault and heading to
http://<ip of server>:8200
Will result in
The token is available to anyone with root access under /root/vault/token/
Add Random passwords to HCP Vaul
The next onehit.yaml Playbook run is:
- name: (One Hit) - Add Services
import_playbook: playbook/add_vault_passwords.yaml
As there is now a secure location available to store passwords for the project, it makes sense to create them and store them in the HCP Vault, when doing so it also makes sense to be generating long random passwords.
This was the biggest area of research as most sites I came across suggested using the hashi_vault collection. the issue I had with this is it uses the lookup function, the lookup function in ansible is only run on the local server from which the ansible code originates from. In my case, this is an AWX box and the code is run on docker images.
To run almost all of the vault related collections HVAC or HVAC Parser needs to be installed for python using PIP. I don't want to be installing this on the awx_task docker each time.
Finally, I came across this page
October 21, 2020
And a wave of obviousness came over me an I'd tested all the API calls using curl locally, Ansible URI using POST was the easiest method of getting my passwords into HCP Vault.
first comes some housekeeping. As some of these commands run from AWX, not the vault host, the URL to the vault is set as the network IP.
There is also a 2-minute pause, without this, I got a random timing issue with Vault which caused the next stage to fail.
- name: (VAULT - Keys) set vault URL as a fact
set_fact:
vault_url: "http://{{ ansible_host}}:8200"
cacheable: yes
- debug: msg="{{ vault_url }}"
- name: Wait 2 minutes
pause:
minutes: "2"
A new secrets store is created in vault
- name: (VAULT - Populate) Create Keystore
environment:
VAULT_TOKEN: "{{ vault_token|quote }}"
VAULT_ADDR: "{{ vault_url|quote }}"
shell: vault secrets enable -version=1 -path "{{ secretstore }}" kv
The URi is then used as a loop over the file services and will create a new secret random password entry (12 characters long) for each item in the services list.
- name: (VAULT - Populate) Add services to Vault
vars:
rndpass: "{{ lookup('password', '/dev/null length=15 chars=ascii_letters,digits') }}"
uri:
url: "{{ vault_url }}/v1/homeserver/{{ item }}"
body_format: json
method: POST
headers:
X-Vault-Token: "{{ vault_token }}"
body: { "options": { "max_versions": "12" }, "data": { "password":"{{ rndpass }}" }}
return_content: yes
status_code: "204"
no_log: True
with_items: "{{ lookup('file', '{{ services }}').splitlines() }}"
Through the WebUI it looks like this
With the password entry looking like this
This results in a populated, HCP Vault instance and the last stage it to consume the passwords.
Using the Vault Data
The last onehit.yaml Playbook run is:
- name: (One Hit) - Testing Role with Dashboard
import_playbook: playbook/install_mariadb.yaml
I used MariaDb for testing as it was a native install and testing is pretty quick
The two important items here are the variable servicename which must match the name of the service listed in the playbook/files/services file and the role vaultpwd
- hosts: all
become_user: root
vars:
servicename: mariadbroot
roles:
- vaultpwd
The role as expected is doing the repetitive heavy lifting.
Much like getting the passwords into the vault it is possible to use the hashi_vault collection, for the same reason I didn't use it before I've not used it here, again I have used the URi builtin
This role needs to be tidied up and more variables to be used to make it more portable.
The data is pulled out of the secure vault in a JSON format
It's not a production password and is long gone
ok: [rhdevhomeoffice] => {
"data.json": {
"auth": null,
"data": {
"data": {
"password": "x9EHkO06KyP9S9A"
},
"options": {
"max_versions": "12"
}
},
"lease_duration": 2764800,
"lease_id": "",
"renewable": false,
"request_id": "ec120c59-54ef-c03e-daad-56b0d14af787",
"warnings": null,
"wrap_info": null
}
}
The data we need is held under data.json.data.data.password and the following code pulls out ONLY the password. I've added debugging to see the relative data
- name: Print returned json dictionary
debug:
var: data.json
- name: Print certain element
debug:
var: data.json.data.data['password']
- name: set_fact some paramater
set_fact:
servicepwd: "{{ data.json.data.data['password'] }}"
The debug data.json line pulls out the data above
To confirm we have the specific element the var: data.json.data.data['password']
Displays
ok: [rhdevhomeoffice] => {
"data.json.data.data['password']": "x9EHkO06KyP9S9A"
}
The important part is setting that data as a fact that can be consumed by the playbook.
ok: [rhdevhomeoffice] => {
"msg": "x9EHkO06KyP9S9A"
}
The variable setpassword is consumed in the example mariadb playbook
- name: Creating DB Creds
copy:
dest: "/root/.my.cnf"
content: |
[client]
user=root
password="{{ servicepwd }}"
and
- name: use "mysql_secure_installation" for Fresh MySQL Installation
mysql_secure_installation:
login_password: ''
new_password: "{{ servicepwd }}"
user: root
login_host: localhost
hosts: ['localhost', '0.0.0.0', '192.168.86.30', '192.168.86.34', '172.17.0.3', '127.0.0.1', '::1']
change_root_password: true
remove_anonymous_user: true
disallow_root_login_remotely: false
remove_test_db: true
register: secure
Once the debug lines are removed the only place the passwords need to be seen during an ansible run are nowhere, they are just consumed.
Git Repo
The code above is available at
Thoughts
While the code isn't perfect, it does work and I'm glad where possible it's native ansible code. This has been a very frustrating project to complete because as a sysadmin I'm not looking to understand the ins and out of JSON parsing I'm just looking to pull a password out of a secure storage area.
I'm pretty sure using AWX has made things more difficult, however in the end it simplified what I needed and reduced the number of collections I need to use. I've also got a better appreciation for roles in Ansible and will look to use them more in the wider code.
What it has taught me, however, again, is that there is no such thing as a simple task when sat down in front of a computer.
References
Sarav AK
October 21, 2020