From Zero to Code: Hashicorp Vault - Using Ansible and AppRole to pull secrets

In this post, I'll be expanding on the previous instructions for installing Hashicorp Vault using Ansible

From Zero to Code: Storing Ansible passwords in Hashicorp Vault

VersionDateDescription1.011 May 2021Initial Post1.220 May 2021the code in theGitLab link at the end of the post is now updated to deploy using HTTPS and usethe API over HTTPS.While working on a personal project and getting to a version1 release, there was an obvious problem with my code (other …

Tech Blog Posts - David FieldDavid Field

While setting up HCP Vault and in the role, I created to prove it was possible to populate the new HCP Vault instance from code I made use of the root/admin token.

While this obviously works, the model promoted within HCP Vault is one of "least privilege".  Giving the least amount of access needed to get something done.

While there is some setup that using the Admin Token can and should be used for, one of the first things which HCP suggest getting setup are roles and policies.

In much the same way you wouldn't run a Linux box as root, you don't run HCP Vault using the admin token.

This post is split into two distinct areas the bash API code using Curl followed by the Ansible implementation and the choices made while doing this.

Things to know

The more astute will notice I'm using --insecure at the end of each curl line. the Dev HCP Vault server is using self-signed certs and I've set up HTTPS access to the API

you may need to install

jq

I've used a couple of variables as well

$VAULT_TOKEN = Random HCP Vault admin token
$VAULT_ADDR = https://127.0.0.1:8200

Reading

The first part of this post is pulled out of the HCP training

AppRole Pull Authentication | Vault - HashiCorp Learn

Learn how to provision, secure, connect, and run any infrastructure for any application.

HashiCorp Learn

From Curl to Secrets

The flow to go from

• Enable AppRole
• Create a Policy to see the Secret store
• Create a role with limited access
• Pull the Details out to need to use the policy and role
• Show secrets accessible

Enable AppRole

 curl --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data '{"type": "approle"}' $VAULT_ADDR/v1/sys/auth/approle --insecure

CREATE a policy

Generate a policy file

This will provide read and update access, but not create

tee payload.json <<"EOF"
{
"policy": "# Read-only permission on 'homeserver/*' path\npath \"homeserver/*\" {\n  capabilities = [ \"read\", \"update\" ]\n}"
}
EOF

Create a policy

curl --header "X-Vault-Token: $VAULT_TOKEN" --request PUT --data @payload.json $VAULT_ADDR/v1/sys/policies/acl/homeserver --insecure

CREATE a role

This file will create a name the token will last 1hr by default

There are additional items

tee rolepayload.json <<EOF
{
"token_policies": "homeserver",
"token_ttl": "1h",
"token_max_ttl": "4h"
}
EOF

create role

curl --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data @rolepayload.json $VAULT_ADDR/v1/auth/approle/role/homeserver --insecure

test role

curl --header "X-Vault-Token: $VAULT_TOKEN" --request GET $VAULT_ADDR/v1/auth/approle/role/homeserver --insecure | jq

GET RoleID and Secret ID

Read the RoleID

curl --header "X-Vault-Token: $VAULT_TOKEN" $VAULT_ADDR/v1/auth/approle/role/homeserver/role-id --insecure | jq

Returns

{
"request_id": "f060d018-4298-3760-4005-d39fea3bd571",
"lease_id": "",
"renewable": false,
"lease_duration": 0,
"data": {
"role_id": "3f54ce6f-07a4-5dd8-11f0-b175094ea4ef"
},
"wrap_info": null,
"warnings": null,
"auth": null
}

Generate a Secret ID, remember this will last about an hour

curl --header "X-Vault-Token: $VAULT_TOKEN" --request POST $VAULT_ADDR/v1/auth/approle/role/homeserver/secret-id --insecure | jq

returns

{
"request_id": "96141158-7867-a52e-694c-e8c48ea04f4e",
"lease_id": "",
"renewable": false,
"lease_duration": 0,
"data": {
"secret_id": "7a0ceffb-41c0-a5e4-6880-df51530fb9b8",
"secret_id_accessor": "ea94b559-d0e4-f320-eacd-72c42712dff7",
"secret_id_ttl": 0
},
"wrap_info": null,
"warnings": null,
"auth": null
}

LOGIN using these creds

Pull the creds out into a json file and from here on out this is an app level token being used.

tee userpayload.json <<"EOF"
{
"role_id": "3f54ce6f-07a4-5dd8-11f0-b175094ea4ef",
"secret_id": "7a0ceffb-41c0-a5e4-6880-df51530fb9b8"
}
EOF

Authenticate to the HCP Vault server using these creds

curl --request POST --data @userpayload.json $VAULT_ADDR/v1/auth/approle/login --insecure | jq

This should return

{
"request_id": "78982acf-abd7-9ca6-5877-ad1769ca7b20",
"lease_id": "",
"renewable": false,
"lease_duration": 0,
"data": null,
"wrap_info": null,
"warnings": null,
"auth": {
"client_token": "s.2oNiabrasc5LRdCa1x9DUf2z",
"accessor": "k7VXQeICA5rTsjKGVjQliVrK",
"policies": [
"default",
"homeserver"
],
"token_policies": [
"default",
"homeserver"
],
"metadata": {
"role_name": "homeserver"
},
"lease_duration": 3600,
"renewable": true,
"entity_id": "179f2cc0-fb0d-390a-3100-3fd1cded2d59",
"token_type": "service",
"orphan": true
}
}

READ Secrets

Use the appRole secret token to pull out the passwords

curl --header "X-Vault-Token: s.2oNiabrasc5LRdCa1x9DUf2z" --request GET $VAULT_ADDR/v1/homeserver/dashboard --insecure | jq

returns

{
"request_id": "bdc9f2b8-0c99-00be-8e09-260c09c3bb34",
"lease_id": "",
"renewable": false,
"lease_duration": 2764800,
"data": {
"data": {
"password": "03TzpMHzNHHIg9K"
},
"options": {
"max_versions": "12"
}
},
"wrap_info": null,
"warnings": null,
"auth": null
}

Which contains the password for Dashboard

"password": "03TzpMHzNHHIg9K"

And if I wait an Hour and try the same thing again

curl --header "X-Vault-Token: s.2oNiabrasc5LRdCa1x9DUf2z" --request GET $VAULT_ADDR/v1/homeserver/dashboard --insecure | jq

Returns

{"errors": ["permission denied"]}

As per the policy and role which was setup.

How to turn this into Ansible?

Remember:

This is not production code, it is littered with debug statements to show what is happening, it would be better split out into a playbook and a role that could be used across other playbooks.  I've also elected to use where possible the built-in Ansible modules rather than the hashi_vault in the Galaxy store.

The purpose of displaying like this to show what can be done.

---
- hosts: localhost
  become_user: root

# https://tg-test100.com/creating-hashicorp-vault-records-with-ansible


  vars:
    vault_url: "https://{{ ansible_host}}:8200"

  roles:
    - gettoken

  tasks:
    - name: Enable the App Role
      command: "vault auth enable approle"
##setup files
    - name: Setup Policy Template
      template:
        dest: /root/hclreadpolicy.json
        src: hclreadpolicy.json.j2
        owner: vault
        group: vault
        mode: '0600'

    - name: Setup Role Temaplate
      template:
        dest: /root/hclroletoken.json
        src: hclroletoken.json.j2
        owner: vault
        group: vault
        mode: '0600'


##import files
    - name: Import RO Policy Temaplate
      uri:
        body_format: json
        validate_certs: false
        headers:
          X-Vault-Token: "{{ vault_token }}"
        url: "{{ vault_url }}/v1/sys/policies/acl/homeserver"
        method: POST
        src: /root/hclreadpolicy.json
        remote_src: yes
        status_code: "204"

    - name: Import Role Temaplate
      uri:
        body_format: json
        validate_certs: false
        headers:
          X-Vault-Token: "{{ vault_token }}"
        url: "{{ vault_url }}/v1/auth/approle/role/homeserver"
        method: POST
        src: /root/hclroletoken.json
        remote_src: yes
        status_code: "204"

##Get RoleID and SecretID

    - name: Get RoleID
      uri: 
        validate_certs: false
        headers:
          X-Vault-Token: "{{ vault_token }}"
        url: "{{ vault_url }}/v1/auth/approle/role/homeserver/role-id"
      register: roleid

    - debug: msg="{{ roleid }}"

    - name: Get RoleID
      uri: 
        body_format: json
        validate_certs: false
        method: POST
        headers:
          X-Vault-Token: "{{ vault_token }}"
        url: "{{ vault_url }}/v1/auth/approle/role/homeserver/secret-id"
      register: secretid

    - debug: msg="{{ secretid }}"

##Pull Out Role ID


    - name: Print returned json dictionary
      debug:
        var: roleid.json

    - name: Print certain element
      debug:
        var: roleid.json.data['role_id']

    - name: set_fact some paramater
      set_fact:
        sessionroleid: "{{ roleid.json.data['role_id'] }}"

    - debug: msg="{{ sessionroleid }}"

##Pull Out secret ID
    - name: Print returned json dictionary
      debug:
        var: secretid.json

    - name: Print certain element
      debug:
        var: secretid.json.data['secret_id']

    - name: set_fact some paramater
      set_fact:
        sessionsecretid: "{{ secretid.json.data['secret_id'] }}"

    - debug: msg="{{ sessionsecretid }}"


##login to pull out approle_token

    - name: Login to Vault to pull access token
      uri:
        body_format: json
        validate_certs: false
        headers:
          X-Vault-Token: "{{ vault_token }}"
        url: "{{ vault_url }}/v1/auth/approle/login"
        method: POST
        body: 
          role_id: "{{ sessionroleid }}"
          secret_id: "{{ sessionsecretid }}"
        status_code: "200"
      register: rotoken

    - debug: msg="{{ rotoken }}"

##Read secrets
    - name: Print returned json dictionary
      debug:
        var: rotoken.json

    - name: Print certain element
      debug:
        var: rotoken.json.auth['client_token']   

    - name: set_fact some paramater
      set_fact:
        rotokenid: "{{ rotoken.json.auth['client_token'] }}"

    - debug: msg="{{ rotokenid }}"

What does this all mean?

The above code can be broken down into 5 key areas

Setting up the ingress files for the policy and role.

  vars:
    vault_url: "https://{{ ansible_host}}:8200"

  roles:
    - gettoken

  tasks:
    - name: Enable the App Role
      command: "vault auth enable approle"
##setup files
    - name: Setup Policy Template
      template:
        dest: /root/hclreadpolicy.json
        src: hclreadpolicy.json.j2
        owner: vault
        group: vault
        mode: '0600'

    - name: Setup Role Temaplate
      template:
        dest: /root/hclroletoken.json
        src: hclroletoken.json.j2
        owner: vault
        group: vault
        mode: '0600'

The First command section Enables the AppRole Authentication method and if viewed in the WebUI this will be seen

The Template sections take the two files I'll use to upload the role and policy settings to the Vault server

hclroletoken.json.j2

{
"token_policies": "homeserver",
"token_ttl": "1h",
"token_max_ttl": "4h"
}

hclreadpolicy.jdon.j2

{
  "policy": "# Read-only permission on 'homeserver/*' path\npath \"homeserver/*\" {\n  capabilities = [ \"read\", \"update\" ]\n}"
}

I've created these as J2 templates because I will have the "homeserver" as a variable moving forward.

The code puts both files in /root and removes the .j2 at the end of the file name.

Importing the Policy and role definitions

##import files
    - name: Import RO Policy Temaplate
      uri:
        body_format: json
        validate_certs: false
        headers:
          X-Vault-Token: "{{ vault_token }}"
        url: "{{ vault_url }}/v1/sys/policies/acl/homeserver"
        method: POST
        src: /root/hclreadpolicy.json
        remote_src: yes
        status_code: "204"

    - name: Import Role Temaplate
      uri:
        body_format: json
        validate_certs: false
        headers:
          X-Vault-Token: "{{ vault_token }}"
        url: "{{ vault_url }}/v1/auth/approle/role/homeserver"
        method: POST
        src: /root/hclroletoken.json
        remote_src: yes
        status_code: "204"

This section ingresses the above files I've moved into /root  into HCP vault and sets up a read only access and a 1hr  before the secret is invalid.

The Policy looks as follows

The token lease expiration can be seen here.

Pulling out the RoleID and SecretID

##Get RoleID and SecretID

    - name: Get RoleID
      uri: 
        validate_certs: false
        headers:
          X-Vault-Token: "{{ vault_token }}"
        url: "{{ vault_url }}/v1/auth/approle/role/homeserver/role-id"
      register: roleid

    - debug: msg="{{ roleid }}"

    - name: Get RoleID
      uri: 
        body_format: json
        validate_certs: false
        method: POST
        headers:
          X-Vault-Token: "{{ vault_token }}"
        url: "{{ vault_url }}/v1/auth/approle/role/homeserver/secret-id"
      register: secretid

    - debug: msg="{{ secretid }}"

When these URi statements are run, they are done so to pull out the role_id and secret_id from the HCP Vault server however what is brought back is a lot of JSON code

role_id variable example

ok: [localhost] => 
  msg:
cache_control: no-store
changed: false
connection: close
content_length: '208'
content_type: application/json
cookies: {}
cookies_string: ''
date: Mon, 24 May 2021 15:28:55 GMT
elapsed: 0
failed: false
json:
  auth: null
  data:
    role_id: 1927d1f1-5114-c72c-5aed-7cdfeb510e02
  lease_duration: 0
  lease_id: ''
  renewable: false
  request_id: 65617dfd-7ca5-8569-82f1-3f0679f0a5b8
  warnings: null
  wrap_info: null
msg: OK (208 bytes)
redirected: false
status: 200
url: https://127.0.0.1:8200/v1/auth/approle/role/homeserver/role-id

The line we want is

   role_id: 1927d1f1-5114-c72c-5aed-7cdfeb510e02

and its the same with the secret URI which you'll notice is a POST not a GET

ok: [localhost] => 
  msg:
cache_control: no-store
changed: false
connection: close
content_length: '288'
content_type: application/json
cookies: {}
cookies_string: ''
date: Mon, 24 May 2021 15:28:55 GMT
elapsed: 0
failed: false
json:
  auth: null
  data:
    secret_id: 098ccae8-6568-fa11-a948-d06bda1db290
    secret_id_accessor: 9d899248-b13c-5b7b-4e9b-0c11cf888b83
    secret_id_ttl: 0
  lease_duration: 0
  lease_id: ''
  renewable: false
  request_id: aff8d444-08d3-4c4e-2a2a-9c3c83aff182
  warnings: null
  wrap_info: null
msg: OK (288 bytes)
redirected: false
status: 200
url: https://127.0.0.1:8200/v1/auth/approle/role/homeserver/secret-id

The line we want here is

secret_id: 098ccae8-6568-fa11-a948-d06bda1db290

Turn the JSON into the fields we need

##Pull Out Role ID


    - name: Print returned json dictionary
      debug:
        var: roleid.json

    - name: Print certain element
      debug:
        var: roleid.json.data['role_id']

    - name: set_fact some paramater
      set_fact:
        sessionroleid: "{{ roleid.json.data['role_id'] }}"

    - debug: msg="{{ sessionroleid }}"

##Pull Out secret ID
    - name: Print returned json dictionary
      debug:
        var: secretid.json

    - name: Print certain element
      debug:
        var: secretid.json.data['secret_id']

    - name: set_fact some paramater
      set_fact:
        sessionsecretid: "{{ secretid.json.data['secret_id'] }}"

    - debug: msg="{{ sessionsecretid }}"

To pull the lines we want as the variables sessionroleid and sessionsecretid a JSON filter is needed

Above is a VERY verbose method of pulling out the data needed from the JSON data the last section provided

The last Debug lines in each section output the following

role_id = sessionroleid

TASK [debug] *******************************************************************
ok: [localhost] => 
  msg: 1927d1f1-5114-c72c-5aed-7cdfeb510e02

secret_id = sessionsecretid

TASK [debug] *******************************************************************
ok: [localhost] => 
  msg: 098ccae8-6568-fa11-a948-d06bda1db290

Pull out the Secret ID to use to read the passwords in Vault

##login to pull out approle_token

    - name: Login to Vault to pull access token
      uri:
        body_format: json
        validate_certs: false
        headers:
          X-Vault-Token: "{{ vault_token }}"
        url: "{{ vault_url }}/v1/auth/approle/login"
        method: POST
        body: 
          role_id: "{{ sessionroleid }}"
          secret_id: "{{ sessionsecretid }}"
        status_code: "200"
      register: rotoken

    - debug: msg="{{ rotoken }}"

##Read secrets
    - name: Print returned json dictionary
      debug:
        var: rotoken.json

    - name: Print certain element
      debug:
        var: rotoken.json.auth['client_token']   

    - name: set_fact some paramater
      set_fact:
        rotokenid: "{{ rotoken.json.auth['client_token'] }}"

    - debug: msg="{{ rotokenid }}"

Once the role_id and secret_id are captured they can be used with a URi POST to approle/login to generate again another JSON stream which looks like this.

ok: [localhost] => 
  msg:
cache_control: no-store
changed: false
connection: close
content_length: '484'
content_type: application/json
cookies: {}
cookies_string: ''
date: Mon, 24 May 2021 15:28:56 GMT
elapsed: 0
failed: false
json:
  auth:
    accessor: DN6V1Ny8mzi9KyPhbYCgvFI0
    client_token: s.8fpZZ58WiYSsC5ghQ9brmwyn
    entity_id: cd029087-02ca-9eb0-8fa2-784153c81638
    lease_duration: 3600
    metadata:
      role_name: homeserver
    orphan: true
    policies:
    - default
    - homeserver
    renewable: true
    token_policies:
    - default
    - homeserver
    token_type: service
  data: null
  lease_duration: 0
  lease_id: ''
  renewable: false
  request_id: 9a3e9552-dbe5-831e-484d-5c4675c07cc8
  warnings: null
  wrap_info: null
msg: OK (484 bytes)
redirected: false
status: 200
url: https://127.0.0.1:8200/v1/auth/approle/login

The line we want is

client_token: s.8fpZZ58WiYSsC5ghQ9brmwyn

Which the remainder of the ansible then pulls out as the read-only token variable rotokenid

TASK [debug] *******************************************************************
ok: [localhost] => 
  msg: s.8fpZZ58WiYSsC5ghQ9brmwyn

This can then be used with other playbooks to pull the passwords in a secure manner out of Hashicorp Vault.

Thoughts

I can't state enough, this code needs work done to it to make it production-ready, when running in the automation tool, in my case Jenkins all these passwords and tokens are plainly visible in the output code.

That's not acceptable and some remove of debug statements and no_log=True need to be added at the very least.

Having all this in 1 playbook is nice, however ideally the last couple of sections should be a role that can be called as needed.

However work it does, using native Ansible modules and it's quick.

Using this as a template it's not a leap of faith to create RW roles as well.

Reference

AppRole - Auth Methods | Vault by HashiCorp

The AppRole auth method allows machines and services to authenticate withVault.

Vault by HashiCorp