Version Date Description
1.0 11 May 2021 Initial Post
1.2 20 May 2021 the code in the GitLab link at the end of the post is now updated to deploy using HTTPS and use the API over HTTPS.

While working on a personal project and getting to a version 1 release, there was an obvious problem with my code (other than it being written by a sysadmin).

There were passwords in the playbooks which even I know is not a good thing.

It’s not good for security, however, it’s also not good for portability or immutability.

The question was how best to store these passwords?

The obvious answer is Ansible Vault however because of the end goal of the project I was looking for something with possibly a WebUI and obviously something which integrated well with Ansible.

Hashicorp Vault was mentioned and so the journey began from having never used Hashicorp Vault to installing it sing Ansible and using it to store passwords and pull them back out.

End Result

The art of moving forward is knowing where you want to get to.. my destination was a simple one. Automate using ansible the following:

  • deploys HCP Vault
  • unseals HCP Vault
  • creates a new secret repository
  • populates the new repository with random secrets for the services which need them
  • has a role to pull the secret out for the service and use it in a playbook.

Essentially going from zero to extracting as-needed service-related random passwords.

Working environment

I have Ansible AWX setup on my home lab and all code is pushed to gitlab, and then run on the remote RHEL8 Dev server from AWX.

This will mean some of the code might need “tweaking” if you just want to run “ansible-playbook” to run this.

Why write this?

Turns out using google to get the output I needed lead me down a lot of rabbit holes to end up with something which worked for me. Hopefully, if you find this it might save you a few Rabbit holes and mistyped Stack Overflow pages.

It’s also notable that many of the pages are written from a developers point of view, something I’m not, so I’ve written this very much from the point of view of a sysadmin.

Disclaimer

Before going ahead with this, let’s deal with the trolls and keyboard warriors who love a bit of a whine because it doesn’t fit their narrow view of the world.  

Not Production Code

I cannot highlight this enough, while this code works, it “works for me” and “works on my dev environment” The point of this post is not to promote this a finished solution. It is a guide to things that may or may not work for you.

become_root

Yes, I know, it’s for development ease of use..

Debugging Enabled

no_log isn’t enabled and the code has several debug statements in it which will display the passwords in the AWX output.

No Guru

I’m learning this, I’m not a code guru, I’m someone who likes a challenge, will use google to find the answers and adapt accordingly to understand what I’m writing.

I’m open to constructive, explained feedback.

Get Started

The file structure is important as Ansible and AWX will not be able to find the files it needs if its in the wrong place

service_deployment  |-> collections  |-> playbook       | -> files          |- **services**       | -> roles          | -> vaultpwd              | -> tasks              | - **main.yaml**       | -> library       | -> templates          | - **config.hcl.j2**       |- **prerequsw.yaml**       |- **install_vault.yaml**       |- **add_vault_passwords.yaml**       |- **install_mariadb.yaml**          |-> **onehit.yaml**

In the root of the Ansible play there is the file I’m using to run this all.

onehit.yaml

---- hosts: all  become: root- name: (One Hit) - Pre Req  import_playbook: playbook/prerequsw.yaml- name: (One Hit) - Install Vault  import_playbook: playbook/install_vault.yaml- name: (One Hit) - Add Services  import_playbook: playbook/add_vault_passwords.yaml- name: (One Hit) - Testing Role with Dashboard  import_playbook: playbook/install_mariadb.yaml

Under playbook there are 4 playbooks

prerequsw.yaml

- hosts: all  become_user: root  tasks:    - name: install PIP3      yum:        name:           - python3          - python3-pip        state: latest

install_vault.yaml

- hosts: all  become_user: root  environment:    VAULT_ADDR: http://127.0.0.1:8200  vars:    vaultdata: /opt/vault/data    vaultip: "0.0.0.0:8200"    unseal_keys_dir_output: /root/vault/keys #fordebug    root_token_dir_output: /root/vault/tokens #fordebug  tasks:  ##Install Prereqs    - name: (VAULT - Pre Reqs) install dependencies      yum:        name: yum-utils        state: latest    - name: (VAULT - Pre Reqs)  pip install ply version 3.8 for hvac[parser]      pip:        name: ply        version: "3.8"        extra_args: --user    - name: (VAULT - Pre Reqs) Install python packages      pip:        name:          - hvac          - hvac[parser]        extra_args: --user##Install Hashicorp Vault    - name: (VAULT - Install) Add repository      yum_repository:        name: hashicorp        description: Hashicorp Repository        baseurl: https://rpm.releases.hashicorp.com/RHEL/$releasever/$basearch/stable        gpgkey: https://rpm.releases.hashicorp.com/gpg    - name: (VAULT - Install) install Hashicorp Vault      yum:        name: vault        state: latest    - name: (VAULT - Install) Create a directory if it does not exist      ansible.builtin.file:        path:           - "{{ vaultdata }}"        state: directory        recurse: yes        owner: vault        group: vault        mode: '0777'    - name: (VAULT - Install) Remove file vault.hcl (delete file)      ansible.builtin.file:        path: /etc/vault.d/vault.hcl        state: absent    - name: (VAULT - Install) Copy Config file      template:        dest: /etc/vault.d/vault.hcl        src: config.hcl.j2        owner: vault        group: vault        mode: '0644'    - name: (VAULT - Install) Start vault service      systemd:        state: restarted        name: vault        enabled: yes        daemon_reload: yes    - name:  (VAULT - Install) Open port 8200      ansible.posix.firewalld:        port: 8200/tcp        permanent: yes        state: enabled    - name: (VAULT - Install) reload service firewalld      systemd:        name: firewalld        state: reloaded## Create Hashicorp Vault keys and token    - name: (VAULT - Keys) Create unseal directories      file:        path: "{{ unseal_keys_dir_output }}"        state: directory    - name: (VAULT - Keys) Create root key directories      file:        path: "{{ root_token_dir_output }}"        state: directory    - name: (VAULT - Keys) Initialise Vault operator      shell: vault operator init -key-shares=5 -key-threshold=3 -format json      environment:        VAULT_ADDR: "http://127.0.0.1:8200"      register: vault_init_results    - name: (VAULT - Keys) Parse output of vault init      set_fact:        vault_init_parsed: "{{ vault_init_results.stdout | from_json }}"    - name: (VAULT - Keys) Write unseal keys to files      copy:        dest: "{{ unseal_keys_dir_output }}/unseal_key_{{ item.0 }}"        content: "{{ item.1 }}"      with_indexed_items: "{{ vault_init_parsed.unseal_keys_hex }}"     - name: (VAULT - Keys) Write root token to file      copy:        content: "{{ vault_init_parsed.root_token }}"        dest: "{{root_token_dir_output}}/rootkey"    - name: (VAULT - Keys) set toot token as fact      set_fact:         vault_token: "{{ vault_init_parsed.root_token }}"        cacheable: yes    - debug: msg="{{ vault_token }}"    - name: (VAULT - Install) Add environmental vars      blockinfile:        path: /etc/environment        block: |          export VAULT_ADDR='http://127.0.0.1:8200'      export VAULT_TOKEN="{{ vault_token }}"    ## unseal vault    - name: (VAULT - Unseal) Reading unseal key contents      command: cat {{item}}      register: unseal_keys      with_fileglob: "{{ unseal_keys_dir_output }}/*"    - name: (VAULT - Unseal) Unseal vault with unseal keys      shell: |        vault operator unseal {{ item.stdout }}      environment:        VAULT_ADDR: "http://127.0.0.1:8200"      with_items: "{{unseal_keys.results}}"

add_vaultpasswords.yaml

---- hosts: all  become_user: root### References  vars:    vault_url: "http://{{ ansible_host}}:8200"    services: "{{ playbook_dir }}/files/services"    secretstore: homeserver  tasks:    - name: (VAULT - Keys) set toot token as fact      set_fact:         vault_url: "http://{{ ansible_host}}:8200"        cacheable: yes    - debug: msg="{{ vault_url }}"    - name: Wait 2 minutes      pause:        minutes: "2"    - name: (VAULT - Populate) Create Keystore      environment:        VAULT_TOKEN: "{{ vault_token|quote }}"        VAULT_ADDR: "{{ vault_url|quote }}"      shell: vault secrets enable -version=1 -path "{{ secretstore }}" kv    - name: (VAULT - Populate) Add services to Vault      vars:        rndpass: "{{ lookup('password', '/dev/null length=15 chars=ascii_letters,digits') }}"      uri:        url: "{{ vault_url }}/v1/homeserver/{{ item }}"        body_format: json        method: POST        headers:          X-Vault-Token: "{{ vault_token }}"        body: { "options": { "max_versions": "12" }, "data": { "password":"{{ rndpass }}" }}        return_content: yes        status_code: "204"      no_log: True      with_items: "{{ lookup('file', '{{ services }}').splitlines() }}"

install_mariadb.yaml

---- hosts: all  become_user: root  vars:    #dbrootpass: CHFEqGGirtddBQjoM6LLy9i6CQK    servicename: mariadbroot  roles:    - vaultpwd  tasks:### Setup MariaDB    - name: DNF - Install MariaDB      dnf:        name:           - mariadb-server          - mariadb        state: present        update_cache: True    - name: FIREWALLD - permit traffic in default zone for mysql service        ansible.posix.firewalld:        service: mysql        permanent: yes        state: enabled    - name: SYSTEMD - Start Mysql Service      service:        name: mariadb        state: started        enabled: true    - name: Creating DB Creds      copy:        dest: "/root/.my.cnf"        content: |          [client]           user=root          password="{{ servicepwd }}"    - import_tasks: mysql-python_dependencies.yml    - name: use "mysql_secure_installation" for Fresh MySQL Installation      mysql_secure_installation:        login_password: ''        new_password: "{{ servicepwd }}"        user: root        login_host: localhost        hosts: ['localhost', '0.0.0.0', '192.168.1.30', '192.168.1.34', '172.17.0.3', '127.0.0.1', '::1']        change_root_password: true        remove_anonymous_user: true        disallow_root_login_remotely: false        remove_test_db: true      register: secure    - debug:        var: secure

Under playbook/templates there are some J2 templates

config.hcl.j2 (referenced in install_vault.yaml)

# Full configuration options can be found at https://www.vaultproject.io/docs/configurationui = true#mlock = true#disable_mlock = truestorage "raft" {  path    = "/opt/vault/data"  node_id = "node1"}# HTTP listenerlistener "tcp" {  address = "0.0.0.0:8200"  tls_disable = 1}# HTTPS listener#listener "tcp" {#  address       = "0.0.0.0:8200"#  tls_cert_file = "/opt/vault/tls/tls.crt"#  tls_key_file  = "/opt/vault/tls/tls.key"#}api_addr = "http://127.0.0.1:8200"cluster_addr = "https://127.0.0.1:8201"

Under playbook/roles there is a simple role

vaultpwd/tasks/main.yaml

- name: (VAULT - Extract) Pull pwd data out of vault  uri:    url: "{{ vault_url }}/v1/homeserver/{{ servicename }}"    body_format: json    method: GET    return_content: yes    headers:      X-Vault-Token: "{{ vault_token }}"  register: data- name: Print returned json dictionary  debug:    var: data.json- name: Print certain element  debug:    var: data.json.data.data['password']- name: set_fact some paramater  set_fact:    servicepwd: "{{ data.json.data.data['password'] }}"- debug: msg="{{ servicepwd }}"

Under files, there is a reference file

playbook/files/services

ldapdashboardkasmuserkasmadminmariadbrootnextclouddbonlyofficedbwgadminwgadminapiwireguard_priv

What’s going on here?

So this is a lot of files, I’ll try and break down my thought process here.

onehit.yaml is the file that kicks all this off and what AWX executes on the remote RHEL8 server.

Install needed packages

The first Playbook run is

- name: (One Hit) - Pre Req  import_playbook: playbook/prerequsw.yaml

Because this is a test branch of the main development there are some prerequisite software missing which needed installing, specifically python.

    - name: install PIP3      yum:        name:           - python3          - python3-pip        state: latest

I have use yum to do the install, longer-term I’ll migrate this to dnf

I’ve tried to keep the python modules needed within the playbooks which need them as I find this easier to debug later.

Install Vault

The next onehit.yaml Playbook run is:

- name: (One Hit) - Install Vault  import_playbook: playbook/install_vault.yaml

This installs vault, it uses the version in the RHEL/Centos Repo’s as directed on the HCP Vault install page

The code needs tidying up however is broken down into the following sections

First, the required software is installed

- name: (VAULT - Pre Reqs) install dependencies  yum:    name: yum-utils    state: latest- name: (VAULT - Pre Reqs)  pip install ply version 3.8 for hvac[parser]  pip:    name: ply    version: "3.8"    extra_args: --user- name: (VAULT - Pre Reqs) Install python packages  pip:    name:      - hvac      - hvac[parser]    #extra_args: --user

The Pip Modules are a requirement of the hashi.vault galaxy collection and I’ve not used this because of a limitation of the AWX version I’m running with lookup functions.

If you’re using the hashi.vault collection then the hvac is needed of the lookups will bork out.

Next Vault is installed

- name: (VAULT - Install) Add repository  yum_repository:    name: hashicorp    description: Hashicorp Repository    baseurl: https://rpm.releases.hashicorp.com/RHEL/$releasever/$basearch/stable    gpgkey: https://rpm.releases.hashicorp.com/gpg- name: (VAULT - Install) install Hashicorp Vault  yum:    name: vault    state: latest

Next, the location of the encrypted data for the vault is created, I’m using locally stored data because of the project this relates to.

The default hcl config file is removed and the j2 template is used to replace it.

It’s important here to make sure that the new vault.hcl is owned by vault and the mode is set correctly or vault will not start.

The J2 in this case is just a text file, it needs the variables added for the listener, API and cluster address added.

- name: (VAULT - Install) Create a directory if it does not exist  ansible.builtin.file:    path:       - "{{ vaultdata }}"    state: directory    recurse: yes    owner: vault    group: vault    mode: '0777'- name: (VAULT - Install) Remove file vault.hcl (delete file)  ansible.builtin.file:    path: /etc/vault.d/vault.hcl    state: absent- name: (VAULT - Install) Copy Config file  template:    dest: /etc/vault.d/vault.hcl    src: config.hcl.j2    owner: vault    group: vault    mode: '0644'

Finally, the vault is started and the required firewall ports opened locally.

- name: (VAULT - Install) Start vault service  systemd:    state: restarted    name: vault    enabled: yes    daemon_reload: yes- name:  (VAULT - Install) Open port 8200  ansible.posix.firewalld:    port: 8200/tcp    permanent: yes    state: enabled- name: (VAULT - Install) reload service firewalld  systemd:    name: firewalld    state: reloaded

with Vault installed, it is however sealed, manually to unseal a Vault instance 5 keys and token are created and 3 of the keys can be used to unseal the vault

The command below is issued

vault operator init

Which would create

Unseal Key 1: 4jYbl2CBIv6SpkKj6Hos9iD32k5RfGkLzlosrrq/JgOmUnseal Key 2: B05G1DRtfYckFV5BbdBvXq0wkK5HFqB9g2jcDmNfTQiSUnseal Key 3: Arig0N9rN9ezkTRo7qTB7gsIZDaonOcc53EHo83F5chAUnseal Key 4: 0cZE0C/gEk3YHaKjIWxhyyfs8REhqkRW/CSXTnmTilv+Unseal Key 5: fYhZOseRgzxmJCmIqUdxEm9C3jB5Q27AowER9w4FC2CkInitial Root Token: s.KkNJYWF5g0pomcCLEmDdOVCW

Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.

Vault does not store the generated master key. Without at least 3 key to
reconstruct the master key, Vault will remain permanently sealed!

It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See “vault operator rekey” for more information.

while looking at how to do this in Ansible there were a lot (a lot) of failed attempts which I think may be down to the AWX docker setup I’m running.

The resulting code runs the vault operator init code and captures the results as a variable, and then parses the results to perform the unseal operation.

In this example, the keys and tokens are held on the server under /root/vault/

- name: (VAULT - Keys) Create unseal directories  file:    path: "{{ unseal_keys_dir_output }}"    state: directory  #delegate_to: localhost- name: (VAULT - Keys) Create root key directories  file:    path: "{{ root_token_dir_output }}"    state: directory- name: (VAULT - Keys) Initialise Vault operator  shell: vault operator init -key-shares=5 -key-threshold=3 -format json  environment:    VAULT_ADDR: "http://127.0.0.1:8200"  register: vault_init_results- name: (VAULT - Keys) Parse output of vault init  set_fact:    vault_init_parsed: "{{ vault_init_results.stdout | from_json }}"- name: (VAULT - Keys) Write unseal keys to files  copy:    dest: "{{ unseal_keys_dir_output }}/unseal_key_{{ item.0 }}"    content: "{{ item.1 }}"  with_indexed_items: "{{ vault_init_parsed.unseal_keys_hex }}" - name: (VAULT - Keys) Write root token to file  copy:    content: "{{ vault_init_parsed.root_token }}"    dest: "{{root_token_dir_output}}/rootkey"- name: (VAULT - Keys) set toot token as fact  set_fact:     vault_token: "{{ vault_init_parsed.root_token }}"    cacheable: yes- debug: msg="{{ vault_token }}"- name: (VAULT - Install) Add environmental vars  blockinfile:    path: /etc/environment    block: |      export VAULT_ADDR='http://127.0.0.1:8200'      export VAULT_TOKEN="{{ vault_token }}"

With the keys separated into their own files, the following ansible code will unlock the vault.  the command vault operator unseal then runs over a loop against the vault server.

- name: (VAULT - Unseal) Reading unseal key contents  command: cat {{item}}  register: unseal_keys  with_fileglob: "{{ unseal_keys_dir_output }}/*"- name: (VAULT - Unseal) Unseal vault with unseal keys  shell: |    vault operator unseal {{ item.stdout }}  environment:    VAULT_ADDR: "http://127.0.0.1:8200"  with_items: "{{unseal_keys.results}}"

This will unlock the vault and heading to

http://<ip of server>:8200

Will result in

The token is available to anyone with root access under /root/vault/token/

Add Random passwords to HCP Vaul

The next onehit.yaml Playbook run is:

- name: (One Hit) - Add Services  import_playbook: playbook/add_vault_passwords.yaml

As there is now a secure location available to store passwords for the project, it makes sense to create them and store them in the HCP Vault, when doing so it also makes sense to be generating long random passwords.

This was the biggest area of research as most sites I came across suggested using the hashi_vault collection. the issue I had with this is it uses the lookup function, the lookup function in ansible is only run on the local server from which the ansible code originates from. In my case, this is an AWX box and the code is run on docker images.

To run almost all of the vault related collections HVAC or HVAC Parser needs to be installed for python using PIP. I don’t want to be installing this on the awx_task docker each time.

Finally, I came across this page

Creating Hashicorp Vault Records with Ansible – TG-TEST100
Using Ansible to automate the creation of credential sets within Hashicorp Vault K/V store

And a wave of obviousness came over me an I’d tested all the API calls using curl locally, Ansible URI using POST was the easiest method of getting my passwords into HCP Vault.

first comes some housekeeping. As some of these commands run from AWX, not the vault host, the URL to the vault is set as the network IP.

There is also a 2-minute pause, without this, I got a random timing issue with Vault which caused the next stage to fail.

- name: (VAULT - Keys) set vault URL as a fact  set_fact:     vault_url: "http://{{ ansible_host}}:8200"    cacheable: yes- debug: msg="{{ vault_url }}"- name: Wait 2 minutes  pause:    minutes: "2"

A new secrets store is created in vault

- name: (VAULT - Populate) Create Keystore  environment:    VAULT_TOKEN: "{{ vault_token|quote }}"    VAULT_ADDR: "{{ vault_url|quote }}"  shell: vault secrets enable -version=1 -path "{{ secretstore }}" kv

The URi is then used as a loop over the file services and will create a new secret random password entry (12 characters long) for each item in the services list.

- name: (VAULT - Populate) Add services to Vault  vars:    rndpass: "{{ lookup('password', '/dev/null length=15 chars=ascii_letters,digits') }}"  uri:    url: "{{ vault_url }}/v1/homeserver/{{ item }}"    body_format: json    method: POST    headers:      X-Vault-Token: "{{ vault_token }}"    body: { "options": { "max_versions": "12" }, "data": { "password":"{{ rndpass }}" }}    return_content: yes    status_code: "204"  no_log: True  with_items: "{{ lookup('file', '{{ services }}').splitlines() }}"

Through the WebUI it looks like this

With the password entry looking like this

This results in a populated, HCP Vault instance and the last stage it to consume the passwords.

Using the Vault Data

The last onehit.yaml Playbook run is:

- name: (One Hit) - Testing Role with Dashboard  import_playbook: playbook/install_mariadb.yaml

I used MariaDb for testing as it was a native install and testing is pretty quick

The two important items here are the variable servicename which must match the name of the service listed in the playbook/files/services file and the role vaultpwd

- hosts: all  become_user: root  vars:    servicename: mariadbroot  roles:    - vaultpwd

The role as expected is doing the repetitive heavy lifting.

Much like getting the passwords into the vault it is possible to use the hashi_vault collection, for the same reason I didn’t use it before I’ve not used it here, again I have used the URi builtin

This role needs to be tidied up and more variables to be used to make it more portable.

The data is pulled out of the secure vault in a JSON format

It’s not a production password and is long gone

ok: [rhdevhomeoffice] => {"data.json": {    "auth": null,    "data": {        "data": {            "password": "x9EHkO06KyP9S9A"        },        "options": {            "max_versions": "12"        }    },    "lease_duration": 2764800,    "lease_id": "",    "renewable": false,    "request_id": "ec120c59-54ef-c03e-daad-56b0d14af787",    "warnings": null,    "wrap_info": null}}

The data we need is held under data.json.data.data.password and the following code pulls out ONLY the password. I’ve added debugging to see the relative data

- name: Print returned json dictionary  debug:    var: data.json- name: Print certain element  debug:    var: data.json.data.data['password']- name: set_fact some paramater  set_fact:    servicepwd: "{{ data.json.data.data['password'] }}"

The debug data.json line pulls out the data above

To confirm we have the specific element the var: data.json.data.data[‘password’]

Displays

ok: [rhdevhomeoffice] => {"data.json.data.data['password']": "x9EHkO06KyP9S9A"}

The important part is setting that data as a fact that can be consumed by the playbook.

ok: [rhdevhomeoffice] => {"msg": "x9EHkO06KyP9S9A"}

The variable setpassword is consumed in the example mariadb playbook

- name: Creating DB Creds  copy:    dest: "/root/.my.cnf"    content: |      [client]       user=root      password="{{ servicepwd }}"

and

- name: use "mysql_secure_installation" for Fresh MySQL Installation  mysql_secure_installation:    login_password: ''    new_password: "{{ servicepwd }}"    user: root    login_host: localhost    hosts: ['localhost', '0.0.0.0', '192.168.86.30', '192.168.86.34', '172.17.0.3', '127.0.0.1', '::1']    change_root_password: true    remove_anonymous_user: true    disallow_root_login_remotely: false    remove_test_db: true  register: secure

Once the debug lines are removed the only place the passwords need to be seen during an ansible run are nowhere, they are just consumed.

Git Repo

The code above is available at

mightywomble_public / ansible_vault
Example non-production code for setting up, populating and consuming Hashicorp Vault

Thoughts

While the code isn’t perfect, it does work and I’m glad where possible it’s native ansible code. This has been a very frustrating project to complete because as a sysadmin I’m not looking to understand the ins and out of JSON parsing I’m just looking to pull a password out of a secure storage area.

I’m pretty sure using AWX has made things more difficult, however in the end it simplified what I needed and reduced the number of collections I need to use. I’ve also got a better appreciation for roles in Ansible and will look to use them more in the wider code.

What it has taught me, however, again, is that there is no such thing as a simple task when sat down in front of a computer.

References

Ansible json_query Examples – Parse JSON from URL response | Devops Junction
ansible json_query examples. How to does ansible json work together. How to Parse JSON data with Ansible and use it as a variable in your playbook. How to get elements of JSON with ansible json_query. ansible json_query is used to parse JSON data and query for the elemtns . Use JSON with ansible. Re…

Creating Hashicorp Vault Records with Ansible – TG-TEST100
Using Ansible to automate the creation of credential sets within Hashicorp Vault K/V store

ansible.builtin.uri – Interacts with webservices — Ansible Documentation

By davidfield

Tech Entusiast