From Zero to Code: Automating Hashicorp Vault with Ansible to create a CA Server

As part of an ongoing Zero to Code blog series, where I take a product I’ve not used before and as a Systems Administrator work through the process of Automating it using Ansible this is another post based on Hashicorp Vault.

Version	Name	Date	Notes
1.0	Initial Post	1 June 2021	Needs work on the code and to be accessible code not yet available.

The first post covered Building HCP Vault using Ansible, adding some passwords and then consuming them.

The second post covered using AppRoles to generate RW and RO tokens for adding and consuming secrets, rather than use the Root token.

This post will cover using the HCP Vault server as an Internal CA server for providing certificates.

As with the previous post, the process will be split into two halves, the first of which is taken straight out of the Hasicorp Tutorial and explains how to do this using the HCP vault API.

The second half of the post will go over the Ansible needed to do the same thing

Why bother doing this?

This is mainly a set of notes in a pretty format for me, it is quick and easy to generate self-signed certs in Ansible on the fly, in fact, that’s what I do to have the HCP Vault server run in HTTPS mode. Having certificates in a central contained place that can act as a CRL, be revoked, removed and generated on the fly is however an important part of certificate management

The Process

The process which is explained looks like this

Disclaimer

This is written from the learning perspective of a Sysadmin NOT a developer, and as such, any code on this page should not be considered production ready.
I am happy to get feedback on the post as long as it’s constructive.
I’m doing all of this using the root token, in a production environment you’d run this using a specific cert_create user, using AppRole or another identity provider.

Things you’ll need

You’ll need an HCP Vault server setup
You’ll need VAULT_TOKEN and VAULT_ADDR variables setup

As an example, this is my /etc/environment

export VAULT_ADDR='https://192.168.40.99:8200'export VAULT_TOKEN="s.udCB3uIQ0hTM9ZZpgJY8nq7W"export VAULT_SKIP_VERIFY=1

Install jq to make the java outputreadable.

Using the HCP Vault API

These instructions are adapted from the excellent tutorial on the Hashicorp site and please refer to the site for reference if you’re having issues here.

Policies

Before the tutorial is started some policies need to be setup on the HCP Vault server

Create following the payload files

tee allmounts.json <<"EOF"{"policy": "# Read-only permission on 'homeserver/' path\npath "sys/mounts/" {\n  capabilities = [ "create", "read", "update", "delete", "list" ]\n}"}EOF

tee somemounts.json <<"EOF"{"policy": "# Read-only permission on 'homeserver/*' path\npath "sys/mounts" {\n  capabilities = [ "read", "list" ]\n}"}EOF

tee pkipayload.json <<"EOF"{"policy": "# Read-only permission on 'homeserver/' path\npath "pki" {\n  capabilities = [ "create", "read", "update", "delete", "list", "sudo" ]\n}"}EOF

Execute the following curl commands to import the templates into the HCP Vault

curl --header "X-Vault-Token: $VAULT_TOKEN" --request PUT --data @allmounts.json $VAULT_ADDR/v1/sys/policies/acl/homeserverpkirw --insecure

curl --header "X-Vault-Token: $VAULT_TOKEN" --request PUT --data @somemounts.json $VAULT_ADDR/v1/sys/policies/acl/homeserverpkiro --insecure

curl --header "X-Vault-Token: $VAULT_TOKEN" --request PUT --data @pkipayload.json $VAULT_ADDR/v1/sys/policies/acl/homeserverpki --insecure

Note I’m using –insecure at the end of the CURL lines because of the self signed cert I’m using.

Enable PKI Secret Store on Vault

Now the policies are set, the next step is to enable the PKI CA secret store in HCP Vault

Create a PKI secret

curl --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data '{"type":"pki"}' $VAULT_ADDR/v1/sys/mounts/pki --insecure

Add a Cert Time to Live (TTL)

curl --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data '{"max_lease_ttl":"87600h"}' $VAULT_ADDR/v1/sys/mounts/pki/tune --insecure

Set Cert details

The common name and the TTL for the CA server needs to be set, this is done again, by creating a payload.

tee commonname.json <<EOF{"common_name": "homeserver.lan","ttl": "87600h"}EOF

Generate the root CA, this will stay on the server in a secure location.

curl --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data @commonname.json $VAULT_ADDR/v1/pki/root/generate/internal --insecure | jq -r ".data.certificate" > CA_cert.crt

Having defined the CA Certificate the Certificate revocation list URL needs to be defined. In this example, the HCP Vault server is being used. Be aware that this URL needs to be reachable from anywhere the client certs are being used. so 127.0.0.1 might not be a good URL.

tee payload-url.json <<EOF{"issuing_certificates": "$VAULT_ADDR/v1/pki/ca","crl_distribution_points": "$VAULT_ADDR/v1/pki/crl"}EOF

This can be pushed to the server

curl --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data @payload-url.json $VAULT_ADDR/v1/pki/config/urls --insecure

The Certificate Authority is now setup

Enable PKI Int server

As the certificates will be generated of an intermediate CA server, this needs to be created in its own separate PKI vault store

Use CURL to create a PKI intermediate store called pki_int

curl --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data '{"type":"pki"}' $VAULT_ADDR/v1/sys/mounts/pki_int --insecuree

Like the CA cert set the TTL for the Int CA cert

curl --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data '{"max_lease_ttl":"43800h"}' $VAULT_ADDR/v1/sys/mounts/pki_int/tune --insecure

Again, set the common name of int cert

tee payload-int.json <<EOF{"common_name": "homeserver.lan Intermediate Authority"}EOF

Generate the intermediate certificate CSR

curl --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data @payload-int.json $VAULT_ADDR/v1/pki_int/intermediate/generate/internal --insecure | jq

This command returns the following:

{"request_id": "4db298bf-1191-5902-5d0a-5db32947f0ad","lease_id": "","renewable": false,"lease_duration": 0,"data": {"csr": "-----BEGIN CERTIFICATE REQUEST-----\nMIICdTCCAV0CAQAwMDEuMCwGA1UEAxMlaG9tZXNlcnZlci5sYW4gSW50ZXJtZWRp\nYXRlIEF1dGhvcml0eTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAOA/\ni9ZvqSlEkcNtbg0xwCvRgwK8y0O4D1HaLghXk0GKbMNeLIl8ryXVRUgSp8FGLj2g\nRN6tHwi0uwaTdLErcyZ34tM3MjQEp47zSS6CbodWskvxcpSEd84hSNnr3BZvmK0e\n8ZPf5Y4r6YjKqL0+rB7N/QZFwG9A3s6BwcgrlT3FOKFr5w7kXCaZY5KpbgIQGR2I\nAGRZqG6/HaG6Z6UkciV1ufLzxsrW0kfAaciwj77d6nbAdbDyd11fNDiZnAOOhGKo\n1b+avcrsbccGEH7i3t+ZUHIbeeNSdpvtDlmo5CKXjv5cnlH3MOEbyKdcRAmlVRPG\njkwH8yFiZYRSgJoqrc0CAwEAAaAAMA0GCSqGSIb3DQEBCwUAA4IBAQBOKAC0utOh\nqKD1kT24wMxOFBW4eDeVv4FyMPj+QhvEjQ9zY59pUpS+aStVX38wI2J81nQZdoNE\n2ICtIqKdR514bLN4koCRdI27UNaji4SKxJttW+dz83CYQon892iOpXPBsdYIep9B\nM3KAsPj2zUzbnmMjQcdUnMbRXjg8DmNuoWG2oZzlmTA95ho09OkBd7ihuE/hAQub\nY9RUjLQTZVzAGfIkBba/hhu5KHqzUe7jLuQtnhY+lsFfd4A8TmmIv5uGADdPWhW/\nWwExDndstD2IKuDmnRl+EMo1gpDfMYRmKggvQ9Cf9Ie1hioeqNGADqwPBaG4caxM\nMF0HDoYVIiyN\n-----END CERTIFICATE REQUEST-----"},"wrap_info": null,"warnings": null,"auth": null}

The CSR is listed in the output and needs to be put into the payload for signing the Intermediate Certificate so the Root CA and the Int CA are trusted.

Sign the intermediate with the CSR by copying the CSR from the above output into the payload for generating the intermediate certificate.

tee payload-int-cert.json <<EOF{"csr": "-----BEGIN CERTIFICATE REQUEST-----\nMIICdTCCAV0CAQAwMDEuMCwGA1UEAxMlaG9tZXNlcnZlci5sYW4gSW50ZXJtZWRp\nYXRlIEF1dGhvcml0eTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAOA/\ni9ZvqSlEkcNtbg0xwCvRgwK8y0O4D1HaLghXk0GKbMNeLIl8ryXVRUgSp8FGLj2g\nRN6tHwi0uwaTdLErcyZ34tM3MjQEp47zSS6CbodWskvxcpSEd84hSNnr3BZvmK0e\n8ZPf5Y4r6YjKqL0+rB7N/QZFwG9A3s6BwcgrlT3FOKFr5w7kXCaZY5KpbgIQGR2I\nAGRZqG6/HaG6Z6UkciV1ufLzxsrW0kfAaciwj77d6nbAdbDyd11fNDiZnAOOhGKo\n1b+avcrsbccGEH7i3t+ZUHIbeeNSdpvtDlmo5CKXjv5cnlH3MOEbyKdcRAmlVRPG\njkwH8yFiZYRSgJoqrc0CAwEAAaAAMA0GCSqGSIb3DQEBCwUAA4IBAQBOKAC0utOh\nqKD1kT24wMxOFBW4eDeVv4FyMPj+QhvEjQ9zY59pUpS+aStVX38wI2J81nQZdoNE\n2ICtIqKdR514bLN4koCRdI27UNaji4SKxJttW+dz83CYQon892iOpXPBsdYIep9B\nM3KAsPj2zUzbnmMjQcdUnMbRXjg8DmNuoWG2oZzlmTA95ho09OkBd7ihuE/hAQub\nY9RUjLQTZVzAGfIkBba/hhu5KHqzUe7jLuQtnhY+lsFfd4A8TmmIv5uGADdPWhW/\nWwExDndstD2IKuDmnRl+EMo1gpDfMYRmKggvQ9Cf9Ie1hioeqNGADqwPBaG4caxM\nMF0HDoYVIiyN\n-----END CERTIFICATE REQUEST-----","format": "pem_bundle","ttl": "43800h"}EOF

use this payload to Sign the Intermediate cert

curl --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data @payload-int-cert.json $VAULT_ADDR/v1/pki/root/sign-intermediate --insecure | jq

This should return the following output

{"request_id": "f3afc412-3a13-01f0-067f-0f84409906f3","lease_id": "","renewable": false,"lease_duration": 0,"data": {"certificate": "-----BEGIN CERTIFICATE-----\nMIIDmzCCAoOgAwIBAgIUZ360AKUYbtI5nlWhGw/4azAUpKIwDQYJKoZIhvcNAQEL\nBQAwADAeFw0yMTA1MjUxNDEwMDlaFw0yNjA1MjQxNDEwMzlaMDAxLjAsBgNVBAMT\nJWhvbWVzZXJ2ZXIubGFuIEludGVybWVkaWF0ZSBBdXRob3JpdHkwggEiMA0GCSqG\nSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDgP4vWb6kpRJHDbW4NMcAr0YMCvMtDuA9R\n2i4IV5NBimzDXiyJfK8l1UVIEqfBRi49oETerR8ItLsGk3SxK3Mmd+LTNzI0BKeO\n80kugm6HVrJL8XKUhHfOIUjZ69wWb5itHvGT3+WOK+mIyqi9Pqwezf0GRcBvQN7O\ngcHIK5U9xTiha+cO5FwmmWOSqW4CEBkdiABkWahuvx2humelJHIldbny88bK1tJH\nwGnIsI++3ep2wHWw8nddXzQ4mZwDjoRiqNW/mr3K7G3HBhB+4t7fmVByG3njUnab\n7Q5ZqOQil47+XJ5R9zDhG8inXEQJpVUTxo5MB/MhYmWEUoCaKq3NAgMBAAGjgdww\ngdkwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFBJt\nxrMi/27z7WHHLZZA0YXJSJGhMB8GA1UdIwQYMBaAFIAqpIrUJYGJXbkm5kKWKwss\nprZ9MD8GCCsGAQUFBwEBBDMwMTAvBggrBgEFBQcwAoYjaHR0cHM6Ly8xOTIuMTY4\nLjg2Ljc6ODIwMC92MS9wa2kvY2EwNQYDVR0fBC4wLDAqoCigJoYkaHR0cHM6Ly8x\nOTIuMTY4Ljg2Ljc6ODIwMC92MS9wa2kvY3JsMA0GCSqGSIb3DQEBCwUAA4IBAQAc\ni7R4E7u8JRVeSf7pTC8Gmz6xc2JNw3hXQ1A5Z+Fl/ZEBLLte/PCW1wgqIau5vhmg\nEdrNZwS8101NGJ+hqJVphQspZjbhPikCqLTmv49swEiYjfPXeYxhBFnYyH5bAFYH\neOcFj2dtF/8UiixgGuAeDcpw2GRhHnDUT3wj5hkoAG0bbLKGU70GJbpA3IAIgyCF\nVT3eZ1ZPphlSYHh9whcvbB4b9Ql0AisPApggQOZNRCrCNzuaVUJqpkxB1zMf4jEm\nEJxqEdp1R34FJi8D2tJNsiwEi0vcsADkYQctMGXmYynNn7n5LcjuWEnh4i9fGqqw\nVs0dXzvYzonunb0PeuMW\n-----END CERTIFICATE-----","expiration": 1779631839,"issuing_ca": "-----BEGIN CERTIFICATE-----\nMIIC8TCCAdmgAwIBAgIUWV3Oji99wO0pfCiUiL4SECv34wswDQYJKoZIhvcNAQEL\nBQAwADAeFw0yMTA1MjUxMzUzMjRaFw0yMTA2MjYxMzUzNTRaMAAwggEiMA0GCSqG\nSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCyocMiu6/l8BHgVhHSdU071ooJP7TRk8NV\n06ODFLYVRoOaGn2EcanAaKslTxKtTkrra2FWuCgDENEuuC4Q0M+5QqXx2qaCXuWT\nSbYZ9VskDSiBBi+mdKDc9Bu1WiKXZ6zKRPCtzw55DTV2A286Q5s/qshIF2dI5KoH\nPgYHQi23ETsrCRsM+jNA9imh4DEPJtqfFqthzcY0ChteMkfkMZp/JLoSvfZaLgKl\npJgagoVUgOP4R7AED2qiUZuQOlVZJr165M/CUQtvEG076IiiqZQglBKxfUFOExR9\nf8oufl0FrpkAH0FIUvXE3eD6CbdwsLj+pQYaAEt/WXvjWkL/JdJrAgMBAAGjYzBh\nMA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBSAKqSK\n1CWBiV25JuZClisLLKa2fTAfBgNVHSMEGDAWgBSAKqSK1CWBiV25JuZClisLLKa2\nfTANBgkqhkiG9w0BAQsFAAOCAQEACA45b7XBVJsF9UZStNNsMghIfTHTiZNSu4aE\nSM2SQTK0JHqlBxdEbbTCqJp0rsuJ7oTBHyXsvyWb1rgHeFhhsj1iYHq7bA8EpSaa\no8INFnm+dTCzNKJSg3JTJ8lO91A5ik8Gk1BdJVF6YvtcpeNLXtdawJzJvoJN5ypo\nl97+679l/pX2nua48Zwc5oJwegXvgPT49HP4HOjiyuc7n2TmEL2uKAOFEDm8qK4O\n8YhTNwatG/uzVb1GYyQ3BURnUY9Bop+zpxrZCQ2jnkigt3S9OiF/xDlBMHEl//QV\n5yLZtgSJEg4DxlnnNeZN6pKAAgaKDShO9bjhxt4CzFLij76ynw==\n-----END CERTIFICATE-----","serial_number": "67:7e:b4:00:a5:18:6e:d2:39:9e:55:a1:1b:0f:f8:6b:30:14:a4:a2"},"wrap_info": null,"warnings": ["The expiration time for the signed certificate is after the CA's expiration time. If the new certificate is not treated as a root, validation paths with the certificate past the issuing CA's expiration time will fail."],"auth": null}

Copy the certificate details out of the above output and included them in the signed certificate payload which will be uploaded to the HCP Vault server.

tee payload-signed.json <<EOF{"certificate": "-----BEGIN CERTIFICATE-----\nMIIDmzCCAoOgAwIBAgIUZ360AKUYbtI5nlWhGw/4azAUpKIwDQYJKoZIhvcNAQEL\nBQAwADAeFw0yMTA1MjUxNDEwMDlaFw0yNjA1MjQxNDEwMzlaMDAxLjAsBgNVBAMT\nJWhvbWVzZXJ2ZXIubGFuIEludGVybWVkaWF0ZSBBdXRob3JpdHkwggEiMA0GCSqG\nSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDgP4vWb6kpRJHDbW4NMcAr0YMCvMtDuA9R\n2i4IV5NBimzDXiyJfK8l1UVIEqfBRi49oETerR8ItLsGk3SxK3Mmd+LTNzI0BKeO\n80kugm6HVrJL8XKUhHfOIUjZ69wWb5itHvGT3+WOK+mIyqi9Pqwezf0GRcBvQN7O\ngcHIK5U9xTiha+cO5FwmmWOSqW4CEBkdiABkWahuvx2humelJHIldbny88bK1tJH\nwGnIsI++3ep2wHWw8nddXzQ4mZwDjoRiqNW/mr3K7G3HBhB+4t7fmVByG3njUnab\n7Q5ZqOQil47+XJ5R9zDhG8inXEQJpVUTxo5MB/MhYmWEUoCaKq3NAgMBAAGjgdww\ngdkwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFBJt\nxrMi/27z7WHHLZZA0YXJSJGhMB8GA1UdIwQYMBaAFIAqpIrUJYGJXbkm5kKWKwss\nprZ9MD8GCCsGAQUFBwEBBDMwMTAvBggrBgEFBQcwAoYjaHR0cHM6Ly8xOTIuMTY4\nLjg2Ljc6ODIwMC92MS9wa2kvY2EwNQYDVR0fBC4wLDAqoCigJoYkaHR0cHM6Ly8x\nOTIuMTY4Ljg2Ljc6ODIwMC92MS9wa2kvY3JsMA0GCSqGSIb3DQEBCwUAA4IBAQAc\ni7R4E7u8JRVeSf7pTC8Gmz6xc2JNw3hXQ1A5Z+Fl/ZEBLLte/PCW1wgqIau5vhmg\nEdrNZwS8101NGJ+hqJVphQspZjbhPikCqLTmv49swEiYjfPXeYxhBFnYyH5bAFYH\neOcFj2dtF/8UiixgGuAeDcpw2GRhHnDUT3wj5hkoAG0bbLKGU70GJbpA3IAIgyCF\nVT3eZ1ZPphlSYHh9whcvbB4b9Ql0AisPApggQOZNRCrCNzuaVUJqpkxB1zMf4jEm\nEJxqEdp1R34FJi8D2tJNsiwEi0vcsADkYQctMGXmYynNn7n5LcjuWEnh4i9fGqqw\nVs0dXzvYzonunb0PeuMW\n-----END CERTIFICATE-----"}EOF

Note: when you copy the certificate from the output to include in the above payload there will be a comma at the end of —–END CERTIFICATE—–“, you’ll need to remove this or the next stage will fail.

And submit the signed intermediate CA certificate back to the HCP Vault server

curl --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data @payload-signed.json $VAULT_ADDR/v1/pki_int/intermediate/set-signed --insecure

Your secrets engine will now look like this with pki and pki_int setup and both having certificates underneath.

Create a role to issue certificates

A role needs to be created to allow clients to request certificates against the INT CA certificate against specific subdomains. In my example, I’ve used the homeserver.lan domain

Create a payload for the certificate role

tee payload-cert-role.json <<EOF{"allowed_domains": "homeserver.lan","allow_subdomains": true,"max_ttl": "720h"}EOF

Push this payload to the HCP Vault server

curl --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data @payload-cert-role.json $VAULT_ADDR/v1/pki_int/roles/homeserver-dot-lan --insecure

Note: In production creating the CA and INT CA certs should be done using their own identity tokens for security purposes. I’d also have separate AppRole (for example) for the Requesting of certificates.

Request client certificates

Having created the CA and Int CA it’s time to generate certificates from the HCP Vault server using the API.

curl --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data '{"common_name": "dashboard.homeserver.lan", "ttl": "30d"}' $VAULT_ADDR/v1/pki_int/issue/homeserver-dot-lan --insecure | jq

There are some things to note here:

“common_name”: “dashboard.homeserver.lan“
“ttl”: “30d“

These can be changed accordingly

The output returned for this is

Example output

{"request_id": "4c9af43d-0c9d-c9ba-1c89-79b85a00a7df","lease_id": "","renewable": false,"lease_duration": 0,"data": {"ca_chain": ["-----BEGIN CERTIFICATE-----\nMIIDmzCCAoOgAwIBAgIUZ360AKUYbtI5nlWhGw/4azAUpKIwDQYJKoZIhvcNAQEL\nBQAwADAeFw0yMTA1MjUxNDEwMDlaFw0yNjA1MjQxNDEwMzlaMDAxLjAsBgNVBAMT\nJWhvbWVzZXJ2ZXIubGFuIEludGVybWVkaWF0ZSBBdXRob3JpdHkwggEiMA0GCSqG\nSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDgP4vWb6kpRJHDbW4NMcAr0YMCvMtDuA9R\n2i4IV5NBimzDXiyJfK8l1UVIEqfBRi49oETerR8ItLsGk3SxK3Mmd+LTNzI0BKeO\n80kugm6HVrJL8XKUhHfOIUjZ69wWb5itHvGT3+WOK+mIyqi9Pqwezf0GRcBvQN7O\ngcHIK5U9xTiha+cO5FwmmWOSqW4CEBkdiABkWahuvx2humelJHIldbny88bK1tJH\nwGnIsI++3ep2wHWw8nddXzQ4mZwDjoRiqNW/mr3K7G3HBhB+4t7fmVByG3njUnab\n7Q5ZqOQil47+XJ5R9zDhG8inXEQJpVUTxo5MB/MhYmWEUoCaKq3NAgMBAAGjgdww\ngdkwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFBJt\nxrMi/27z7WHHLZZA0YXJSJGhMB8GA1UdIwQYMBaAFIAqpIrUJYGJXbkm5kKWKwss\nprZ9MD8GCCsGAQUFBwEBBDMwMTAvBggrBgEFBQcwAoYjaHR0cHM6Ly8xOTIuMTY4\nLjg2Ljc6ODIwMC92MS9wa2kvY2EwNQYDVR0fBC4wLDAqoCigJoYkaHR0cHM6Ly8x\nOTIuMTY4Ljg2Ljc6ODIwMC92MS9wa2kvY3JsMA0GCSqGSIb3DQEBCwUAA4IBAQAc\ni7R4E7u8JRVeSf7pTC8Gmz6xc2JNw3hXQ1A5Z+Fl/ZEBLLte/PCW1wgqIau5vhmg\nEdrNZwS8101NGJ+hqJVphQspZjbhPikCqLTmv49swEiYjfPXeYxhBFnYyH5bAFYH\neOcFj2dtF/8UiixgGuAeDcpw2GRhHnDUT3wj5hkoAG0bbLKGU70GJbpA3IAIgyCF\nVT3eZ1ZPphlSYHh9whcvbB4b9Ql0AisPApggQOZNRCrCNzuaVUJqpkxB1zMf4jEm\nEJxqEdp1R34FJi8D2tJNsiwEi0vcsADkYQctMGXmYynNn7n5LcjuWEnh4i9fGqqw\nVs0dXzvYzonunb0PeuMW\n-----END CERTIFICATE-----"],"certificate": "-----BEGIN CERTIFICATE-----\nMIIDeTCCAmGgAwIBAgIUdfFGkvjR8NBgihzsnvmiP4kWQX8wDQYJKoZIhvcNAQEL\nBQAwMDEuMCwGA1UEAxMlaG9tZXNlcnZlci5sYW4gSW50ZXJtZWRpYXRlIEF1dGhv\ncml0eTAeFw0yMTA1MjUxNDI5MDRaFw0yMTA1MjYxNDI5MzNaMCMxITAfBgNVBAMT\nGGRhc2hib2FyZC5ob21lc2VydmVyLmxhbjCCASIwDQYJKoZIhvcNAQEBBQADggEP\nADCCAQoCggEBANg1WOU1EL7ifQ+KthWJTZOMrIoT+OBro+dFZwlL56Y7F8znwnFF\nw1yy/yD6R7FmzE2oNnVnaHBmJBA91dm8m9wQ5z98M9d7Ir/nABechUzrGiiI0O5t\n5DqMzJxiUqyklvXsX0xnAxbH3X5DQ0PcKRUVMBHARDeV05efm2VIEzdVn9O4f5RG\n1L2FfBO258CtGOQYqoW73WggxVwdsLAg6OGm6BrO0kgDXugsQyeEVq+jd71ou6NV\nRac2fqFkv5B+MDcQ64y1+1YIdju7JlrWWjI7v38KkYUxKeuTP7+eCoEDQLivLDIm\nfmdtol8TihiWhH8p+Hb6+OnjxJt+dEK2NJECAwEAAaOBlzCBlDAOBgNVHQ8BAf8E\nBAMCA6gwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMB0GA1UdDgQWBBQk\nEh5MLrSOsG951cJzyZCWtdxHIjAfBgNVHSMEGDAWgBQSbcazIv9u8+1hxy2WQNGF\nyUiRoTAjBgNVHREEHDAaghhkYXNoYm9hcmQuaG9tZXNlcnZlci5sYW4wDQYJKoZI\nhvcNAQELBQADggEBALHvdhRIpIyWmdlaw6zsLQSTu3kMoJ7HBoM87JbYGtsVaiDA\n3f4Sc//yoKKpHWPsnePjVit81hBoiyTz0mwlFWjFKRBaCsctb16oynox2J5YUcBL\nzChbZ0BEAVmpcgVKRs43g70nCy12Nq0LiyOhYx918OvVyxsUV3sBgg6H7Lms02o9\nEMde3/O8rbnbRU90SV0bYPXRvh6gzErOtJGdK/tc3Djle6SUuhQV/zT+Zg4V1mAf\nwMUPr4GZku5wA7cgDAjUCQGyl8A0OtwTYUnks8d/I7osN8nV2kVhD/rMLp5saqGy\nLtaGIB2bBiedsw7BP+n3SCdMm6ePik+052kjFxg=\n-----END CERTIFICATE-----","expiration": 1622039373,"issuing_ca": "-----BEGIN CERTIFICATE-----\nMIIDmzCCAoOgAwIBAgIUZ360AKUYbtI5nlWhGw/4azAUpKIwDQYJKoZIhvcNAQEL\nBQAwADAeFw0yMTA1MjUxNDEwMDlaFw0yNjA1MjQxNDEwMzlaMDAxLjAsBgNVBAMT\nJWhvbWVzZXJ2ZXIubGFuIEludGVybWVkaWF0ZSBBdXRob3JpdHkwggEiMA0GCSqG\nSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDgP4vWb6kpRJHDbW4NMcAr0YMCvMtDuA9R\n2i4IV5NBimzDXiyJfK8l1UVIEqfBRi49oETerR8ItLsGk3SxK3Mmd+LTNzI0BKeO\n80kugm6HVrJL8XKUhHfOIUjZ69wWb5itHvGT3+WOK+mIyqi9Pqwezf0GRcBvQN7O\ngcHIK5U9xTiha+cO5FwmmWOSqW4CEBkdiABkWahuvx2humelJHIldbny88bK1tJH\nwGnIsI++3ep2wHWw8nddXzQ4mZwDjoRiqNW/mr3K7G3HBhB+4t7fmVByG3njUnab\n7Q5ZqOQil47+XJ5R9zDhG8inXEQJpVUTxo5MB/MhYmWEUoCaKq3NAgMBAAGjgdww\ngdkwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFBJt\nxrMi/27z7WHHLZZA0YXJSJGhMB8GA1UdIwQYMBaAFIAqpIrUJYGJXbkm5kKWKwss\nprZ9MD8GCCsGAQUFBwEBBDMwMTAvBggrBgEFBQcwAoYjaHR0cHM6Ly8xOTIuMTY4\nLjg2Ljc6ODIwMC92MS9wa2kvY2EwNQYDVR0fBC4wLDAqoCigJoYkaHR0cHM6Ly8x\nOTIuMTY4Ljg2Ljc6ODIwMC92MS9wa2kvY3JsMA0GCSqGSIb3DQEBCwUAA4IBAQAc\ni7R4E7u8JRVeSf7pTC8Gmz6xc2JNw3hXQ1A5Z+Fl/ZEBLLte/PCW1wgqIau5vhmg\nEdrNZwS8101NGJ+hqJVphQspZjbhPikCqLTmv49swEiYjfPXeYxhBFnYyH5bAFYH\neOcFj2dtF/8UiixgGuAeDcpw2GRhHnDUT3wj5hkoAG0bbLKGU70GJbpA3IAIgyCF\nVT3eZ1ZPphlSYHh9whcvbB4b9Ql0AisPApggQOZNRCrCNzuaVUJqpkxB1zMf4jEm\nEJxqEdp1R34FJi8D2tJNsiwEi0vcsADkYQctMGXmYynNn7n5LcjuWEnh4i9fGqqw\nVs0dXzvYzonunb0PeuMW\n-----END CERTIFICATE-----","private_key": "-----BEGIN RSA PRIVATE KEY-----\nMIIEpAIBAAKCAQEA2DVY5TUQvuJ9D4q2FYlNk4ysihP44Guj50VnCUvnpjsXzOfC\ncUXDXLL/IPpHsWbMTag2dWdocGYkED3V2byb3BDnP3wz13siv+cAF5yFTOsaKIjQ\n7m3kOozMnGJSrKSW9exfTGcDFsfdfkNDQ9wpFRUwEcBEN5XTl5+bZUgTN1Wf07h/\nlEbUvYV8E7bnwK0Y5BiqhbvdaCDFXB2wsCDo4aboGs7SSANe6CxDJ4RWr6N3vWi7\no1VFpzZ+oWS/kH4wNxDrjLX7Vgh2O7smWtZaMju/fwqRhTEp65M/v54KgQNAuK8s\nMiZ+Z22iXxOKGJaEfyn4dvr46ePEm350QrY0kQIDAQABAoIBADNm83yC0jlfpXX1\nd3bFTvE+Z6LoPqo0TSJlyKpYJnnJ4M2xZ/QALwMx9yADANp3YykvTcs5y4W1cut9\nmAMNKUz3o9LfF4AqYUeYhtgWOUbhOjXa2TlmXPVilh6z7Y3oD4/mI34Jm51l5Q3o\ntexDQm0lvWjq+gzxDP4mTw6URSVJQNa1L2rF0dQoDzl9nkTZJTj/YW7giawNAmLv\nyhsKKflPkj/XImk73kqEEkWLmV3/TFSQktcEeO1Zfljg8MuHQe7P3ulI8iYWiAQj\nBX4RM6wGnH+nXWa3htnz9OiTAYVdITPNm/7991syF7sPkCOW3D7kN2CjI3GGyOGS\n19gWK9UCgYEA/6AdFj9oROxWYWYziFfi53Tsq3dURr5pUvCJxUJdGHLLCcIPKA0d\n97xMHxM5jFK1P/3Uo+iine0iivDhxllZpZQ3m02Z42NPxp+9CN9K21Zjf+0ZWRdM\n633vhMO3wovOnFwdhml2r9lRpbt+GkAdZiHzHPgxyL8okmamYEnaQAcCgYEA2IZy\nti0z7G9X9cQwhNA6XIMXhGI3RkbDJ5CXodck6razEB+lr0BtsyvnB1lZxr0RzONd\nfP7pRTKN3pZAVnMQJN2sJrVPugxwp2qFPmhLLOJphCEVwqwU0Z6Bn5o5FovDDOAp\ne6WXodh/igCj05dcJ55R4XpwmEZMJ1cSpSJ6EKcCgYEAuLX9zqGquoL8OA0dl5vJ\n/e3jRlNHtobInIHrS3qUwqHQTRDI2uv/h4+sgZfmsZriFYdZK8diGjPMDhHZUvYl\nbRwYwkPkuwZ8Es5CTjLraGqYI0w0UMghcNjjRlAWbKGRfjKhswpqFM83zEYa7OT/\nWVmWzowZjTF0I7XA6zryVekCgYEAhTi77bEESI39Xb63Z5BCyFb0KkTP45J4UqiZ\nUz9vfGaq59nA9II8vMffXtsv7KK6CAlApT6mQignt/NUZJxpK3WkjTBzfHJZAfj9\nQHelAVnRODWvENcV/B99e7jFNUUK3qoxe91X3YG6fyuDoRV44vt7P7M5AcgG5RGi\n7C25UvMCgYAJ/PXfgQ5+2+GujAz0KLgYqdxEvMsY8XaHx+zmnuYb30HK6CxQ3kG1\nHzt6Itd+sjNasuVBWLrnTEku4RB66u7rv0WEJ1LDo8CaLE3ztIfTMW0dqCu+OOnd\n23vhE0aW3spEPiBzDgU94oZ+jN+cGhN6+AhL7SRPL69zj1O/Qhn5fQ==\n-----END RSA PRIVATE KEY-----","private_key_type": "rsa","serial_number": "75:f1:46:92:f8:d1:f0:d0:60:8a:1c:ec:9e:f9:a2:3f:89:16:41:7f"},"wrap_info": null,"warnings": null,"auth": null}

Dump each of these certificates into files

cachain.pem
issuing.pem
certificate.pem
key.pem

Additional Features

As well as creating certificates you’ll need to revoke and remove certs if they are compromised at the start of this a CRL link back to the HCP Vault server was added to the CA and as such any cert created which can be traced back to that CA cert will know where to look for revoked certificates.

Revoke Certs

curl --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data '{"serial_number": "3f:03:a9:10:e0:28:df:dc:d8:a8:f9:50:7b:cd:d4:8c:01:f5:47:5e"}' $VAULT_ADDR/v1/pki_int/revoke --insecure

Remove Expired certs

curl --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data '{"tidy_cert_store": true, "tidy_revoked_certs": true}' $VAULT_ADDR/v1/pki_int/tidy --insecure

Summary

This is how to set up a CA server using the API on HCP Vault, the next step is to automate this using Ansible. There are lots more features that can be used here, this is a quick high-level overview.

Using Ansible

The idea here is to make the API calls above and use them in Ansible using the URi builtin to create an automated deployment of a CA server in HCP Vault.

Ansible Code

The Ansible code for this up until 1 point was pretty simple and ended up looking like this.

Warning: formatting WILL be off on this if you copy and paste it

##Playbook to deploy aCA Server on Vault##It needs vault setting up with RW app roles first---- hosts: localhost  become_user: root  vars:    vault_url: "https://{{ ansible_host}}:8200"    mydomain: homeserver.lan  roles:    - getrwtoken    - gettoken  tasks:## Create Policy Paylod files from Templates    - name: (PKI Server) Create All Mounts Policy      template:        dest: /root/caallmounts.json        src: ca/allmounts.json.j2        owner: vault        group: vault        mode: '0600'    - name: (PKI Server) Create Some Mounts Policy      template:        dest: /root/casomemounts.json        src: ca/somemounts.json.j2        owner: vault        group: vault        mode: '0600'    - name: (PKI Server) Create PKI Policy Policy      template:        dest: /root/capkipayload.json        src: ca/pkipayload.json.j2        owner: vault        group: vault        mode: '0600' ## Import JSON policy Files into HCP Vault    - name: (PKI Server) Import All Mounts Policy      uri:        body_format: json        validate_certs: false        headers:          X-Vault-Token: "{{ vault_token }}"        url: "{{ vault_url }}/v1/sys/policies/acl/{{ item }}"        method: POST        src: "/root/{{ item }}.json"        remote_src: yes        status_code: "204"      with_items:        - caallmounts        - casomemounts    - capkipayload## Create the PKI secret    - name: (PKI server) Create PKI CA Store      uri:        body_format: json        validate_certs: false        headers:          X-Vault-Token: "{{ vault_token }}"        url: "{{ vault_url }}/v1/sys/mounts/pki"        method: POST        body:           type: "pki"        status_code: "204"    - name: (PKI server) Create PKI CA Store TTL      uri:        body_format: json        validate_certs: false        headers:          X-Vault-Token: "{{ vault_token }}"        url: "{{ vault_url }}/v1/sys/mounts/pki/tune"        method: POST        body:           max_lease_ttl: "87600h"          status_code: "204"    - name: (PKI Server) Create PKI Common Name Policy      template:        dest: /root/commonname.json        src: ca/commonname.json.j2        owner: vault        group: vault        mode: '0600'## Generate CA cert- name: (PKI Server) Generate CA cert  uri:    body_format: json    validate_certs: false    headers:      X-Vault-Token: "{{ vault_token }}"    url: "{{ vault_url }}/v1/pki/root/generate/internal"    method: POST    src: "/root/commonname.json"    remote_src: yes    status_code: "200"  register: certdata- name: set_fact some paramater  set_fact:    cacert: "{{ certdata.json.data['certificate'] }}"- debug: msg="{{ cacert }}"##set CRL- name: (PKI Server) Create CRL Policy File  template:    dest: /root/payload-url.json    src: ca/payload-url.json.j2    owner: vault    group: vault    mode: '0600'- name: (PKI Server) Import CRL Policy  uri:    body_format: json    validate_certs: false    headers:      X-Vault-Token: "{{ vault_token }}"    url: "{{ vault_url }}/v1/pki/config/urls"    method: POST    src: "/root/{{ item }}.json"    remote_src: yes    status_code: "204"  with_items:    - payload-url##CREATE PKI INIT SERVER- name: (PKI server) Create PKI Int CA Store  uri:    body_format: json    validate_certs: false    headers:      X-Vault-Token: "{{ vault_token }}"    url: "{{ vault_url }}/v1/sys/mounts/pki_int"    method: POST    body:       type: "pki"    status_code: "204"- name: (PKI server) Create PKI Int CA Store TTL  uri:    body_format: json    validate_certs: false    headers:      X-Vault-Token: "{{ vault_token }}"    url: "{{ vault_url }}/v1/sys/mounts/pki_int/tune"    method: POST    body:       max_lease_ttl: "43800h"      status_code: "204"- name: (PKI Server) Create PKI Int Common Name Policy File  template:    dest: /root/payload-int.json    src: ca/payload-int.json.j2    owner: vault    group: vault    mode: '0600'##create CSR- name: (PKI Server) Generate PKI Int CSR   uri:    body_format: json    validate_certs: false    headers:      X-Vault-Token: "{{ vault_token }}"    url: "{{ vault_url }}/v1/pki_int/intermediate/generate/internal"    method: POST    src: "/root/payload-int.json"    remote_src: yes    status_code: "200"  register: intcsrdata- debug: msg="{{ intcsrdata }}"- name: Pull out just the CSR as a fact  set_fact:    csrcert: "{{ intcsrdata.json.data['csr'] | trim }}"- debug: msg="{{ csrcert }}"## Sign Int CA Cert with CSR- name: Ansible PKI Int Cert CSR Payload  copy:    dest: "/root/payload-csr-cert.json"    content: |      {        "csr": "{{ csrcert }}",        "format": "pem_bundle",        "ttl": "43800h"      }################################################## Need to add newline \n to the end of the first line### the beginning and end of the last line### https://groups.google.com/g/vault-tool/c/vaBtI-S6f14/m/G5blRvx3AQAJ###############################################- name: add newline at start of cert  ansible.builtin.replace:    path: /root/payload-csr-cert.json    regexp: '-----BEGIN CERTIFICATE REQUEST-----'    replace: '-----BEGIN CERTIFICATE REQUEST-----\\n'- name: add 2 newlines at the end of the cert  ansible.builtin.replace:    path: /root/payload-csr-cert.json    regexp: '-----END CERTIFICATE REQUEST-----'    replace: '\\n-----END CERTIFICATE REQUEST-----\\n'- name: Change file ownership, group and permissions  ansible.builtin.file:    path: /root/payload-csr-cert.json    owner: vault    group: vault    mode: '0644'

This almost (as I’ll explain below in “the final bit” ) ran the whole process as a playbook to setup a CA server on HCP Vault.

Code Breakdown

Some quick notes

I use validate_certs: false in the URL because the Vault was set up using a self-signed cert
The YAML formatting might be off off
I need to add several creates to this code to make it immutable.

Setting files, ownership and rights

There are several files I’veincluded as j2 templates to allow them to be expanded as I improve this code and streamline it, which I’ve moved into the /root folder. Set the permissions so they are readable by Vault.

## Create Policy Paylod files from Templates    - name: (PKI Server) Create All Mounts Policy      template:        dest: /root/caallmounts.json        src: ca/allmounts.json.j2        owner: vault        group: vault        mode: '0600'    - name: (PKI Server) Create Some Mounts Policy      template:        dest: /root/casomemounts.json        src: ca/somemounts.json.j2        owner: vault        group: vault        mode: '0600'    - name: (PKI Server) Create PKI Policy Policy      template:        dest: /root/capkipayload.json        src: ca/pkipayload.json.j2        owner: vault        group: vault        mode: '0600'          - name: (PKI Server) Create PKI Common Name Policy      template:        dest: /root/commonname.json        src: ca/commonname.json.j2        owner: vault        group: vault        mode: '0600'   - name: (PKI Server) Create CRL Policy File      template:        dest: /root/payload-url.json        src: ca/payload-url.json.j2        owner: vault        group: vault        mode: '0600'

These files are called by the various URi POST and GET calls

caallmounts.json

{"policy": "# Read-only permission on 'homeserver/' path\npath \"sys/mounts/*\" {\n  capabilities = [ \"create\", \"read\", \"update\", \"delete\", \"list\" ]\n}"}

casomemounts.json

{"policy": "# Read-only permission on 'homeserver/*' path\npath \"sys/mounts\" {\n  capabilities = [ \"read\", \"list\" ]\n}"}

capkipayload.json

{"policy": "# Read-only permission on 'homeserver/*' path\npath \"pki*\" {\n  capabilities = [ \"create\", \"read\", \"update\", \"delete\", \"list\", \"sudo\" ]\n}"}

commonname.json

{"common_name": "homeserver.lan","ttl": "87600h"}

payload-url.json

{  "issuing_certificates": "https://127.0.0.1:8200/v1/pki/ca",  "crl_distribution_points": "https://127.0.0.1:8200/v1/pki/crl"}

A quick note on this last one, for the CRL to work, this needs to be a URL accessible by any endpoint using certs generated by the Certificate Authority

It should look something like this if the public URL of your Vault server was 192.168.99.7

{  "issuing_certificates": "https://192.168.99.7:8200/v1/pki/ca",  "crl_distribution_points": "https://192.168.99.7:8200/v1/pki/crl"}

Create a HCP Vault Policy

 ## Import JSON policy Files into HCP Vault    - name: (PKI Server) Import All Mounts Policy      uri:        body_format: json        validate_certs: false        headers:          X-Vault-Token: "{{ vault_token }}"        url: "{{ vault_url }}/v1/sys/policies/acl/{{ item }}"        method: POST        src: "/root/{{ item }}.json"        remote_src: yes        status_code: "204"      with_items:        - caallmounts        - casomemounts        - capkipayload

This play loops around the above 3 policy files and imports them into Vault

Setup the PKI Space

As we have set up Secrets before and needed to enable the KV Engine the PKI engine needs to be enabled.

## Create the PKI secret    - name: (PKI server) Create PKI CA Store      uri:        body_format: json        validate_certs: false        headers:          X-Vault-Token: "{{ vault_token }}"        url: "{{ vault_url }}/v1/sys/mounts/pki"        method: POST        body:           type: "pki"        status_code: "204"    - name: (PKI server) Create PKI CA Store TTL      uri:        body_format: json        validate_certs: false        headers:          X-Vault-Token: "{{ vault_token }}"        url: "{{ vault_url }}/v1/sys/mounts/pki/tune"        method: POST        body:           max_lease_ttl: "87600h"          status_code: "204"

The core root CA store is also created.

Generate the Certificate Authority (CA) Root Cert

Once the PKI CA Engine is running, the commonname.json file is used to create a root certificate.

 ## Generate CA cert- name: (PKI Server) Generate CA cert  uri:    body_format: json    validate_certs: false    headers:      X-Vault-Token: "{{ vault_token }}"    url: "{{ vault_url }}/v1/pki/root/generate/internal"    method: POST    src: "/root/commonname.json"    remote_src: yes    status_code: "200"  register: certdata- name: set_fact some paramater  set_fact:    cacert: "{{ certdata.json.data['certificate'] }}"- debug: msg="{{ cacert }}"

Ansible JSON formatting is used to extract only the certificate details from the output and store this as a variable cacert.

Create CRL Policy

With the CA server running, the payload-url.json file is used to ensure certificates can be revoked and checked to see if they are still valid.

   - name: (PKI Server) Import CRL Policy     uri:       body_format: json       validate_certs: false       headers:         X-Vault-Token: "{{ vault_token }}"         url: "{{ vault_url }}/v1/pki/config/urls"       method: POST       src: "/root/{{ item }}.json"       remote_src: yes       status_code: "204"     with_items:       - payload-url

This can be checked and edited by navigating to:

Secrets -> PKI -> Configuration -> Configure -> URLs

Which will display

Create the Int-CA Server on HCP Vault

Because the Root CA should be kept separate with strict policies around access, an Intermediate CA is created as a separate PKI Engine.

##CREATE PKI INIT SERVER- name: (PKI server) Create PKI Int CA Store  uri:    body_format: json    validate_certs: false    headers:      X-Vault-Token: "{{ vault_token }}"    url: "{{ vault_url }}/v1/sys/mounts/pki_int"    method: POST    body:       type: "pki"    status_code: "204"- name: (PKI server) Create PKI Int CA Store TTL  uri:    body_format: json    validate_certs: false    headers:      X-Vault-Token: "{{ vault_token }}"    url: "{{ vault_url }}/v1/sys/mounts/pki_int/tune"    method: POST    body:       max_lease_ttl: "43800h"      status_code: "204"

This is set up in much the same way as the Rook CA PKI Engine

Create CSR

A certificate signing request needs to be created to have the Int-CA cert signed by the CA cert for authenticity.

##create CSR- name: (PKI Server) Generate PKI Int CSR   uri:    body_format: json    validate_certs: false    headers:      X-Vault-Token: "{{ vault_token }}"    url: "{{ vault_url }}/v1/pki_int/intermediate/generate/internal"    method: POST    src: "/root/payload-int.json"    remote_src: yes    status_code: "200"  register: intcsrdata- debug: msg="{{ intcsrdata }}"- name: Pull out just the CSR as a fact  set_fact:    csrcert: "{{ intcsrdata.json.data['csr'] | trim }}"- debug: msg="{{ csrcert }}"

This will use payload-int.json which will define which domain this intermediate server will sign certificates for.

{  "common_name": "homeserver.lan Intermediate Authority"}

The resulting output is filtered using a JSON Query to pull out only the CSR information needed into a variable csrcert

Create CSR Payload File

The CSR needs to output to a file so the URi can ingress it into a file (more on this after this section)

## Sign Int CA Cert with CSR- name: Ansible PKI Int Cert CSR Payload  copy:    dest: "/root/payload-csr-cert.json"    content: |      {        "csr": "{{ csrcert }}",        "format": "pem_bundle",        "ttl": "43800h"      }################################################## Need to add newline \n to the end of the first line### the beginning and end of the last line### https://groups.google.com/g/vault-tool/c/vaBtI-S6f14/m/G5blRvx3AQAJ###############################################- name: add newline at start of cert  ansible.builtin.replace:    path: /root/payload-csr-cert.json    regexp: '-----BEGIN CERTIFICATE REQUEST-----'    replace: '-----BEGIN CERTIFICATE REQUEST-----\\n'- name: add 2 newlines at the end of the cert  ansible.builtin.replace:    path: /root/payload-csr-cert.json    regexp: '-----END CERTIFICATE REQUEST-----'    replace: '\\n-----END CERTIFICATE REQUEST-----\\n'- name: Change file ownership, group and permissions  ansible.builtin.file:    path: /root/payload-csr-cert.json    owner: vault    group: vault    mode: '0644'

Problems

At this point, I started having issues with Ansible and a URi Post. No matter how i formatted the resulting JSON file it would not import using ansible but would using the curl command from the command line.

The resulting file looked like this

{  "csr": "-----BEGIN CERTIFICATE REQUEST-----\nMIICdTCCAV0CAQAwMDEuMCwGA1UEAxMlaG9tZXNlcnZlci5sYW4gSW50ZXJtZWRpYXRlIEF1dGhvcml0eTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBANi0QEL0hw4jlf+iCo3Oe/AHP3h/Nd06cLbJ+DYCWpmrLTRT1Tp/e+k7S8cp3keN5j4pJzw53Io8JoUDVJF5vtNDfNLBt4cMn6kHv1cpYj+8mPOg8ZdxjVaYz3kKaVIH3Myna68tD7FJwekvs081SWQ+s6dog0jXefipzx+0V2aCu7Pe+RzYxfIM9XXqe8C4/5EoRvpropiw0ewRC9QSfD9CbnkTs1VzAx4q3on7db4Rvv2ZPoXOdcYf+AyYGb239C3wp8CTeQAT+wcELbUev1aRpXF55ZCrwIx+kQvMNUQDIGWDsWDUXFILgIKc+P+WGWjiqB1OGg0uuCM2NaGT/10CAwEAAaAAMA0GCSqGSIb3DQEBCwUAA4IBAQCuIDGTc7XUVqlj7mS0HYJXuX5WFWknsBTwcRolcN/4rLB57AmF+a1Hj4nc7tvwuQBXP9N+B6hTV8IgxRb2nduTS34Ap5jaFa8lzQo6cbug61LlUJ/j956IH/2i/Y1kgLAvhkhkoy6chREUE4UTOLNZsWbzXOYovrHNpJqIvFstanHeruImbUj5Ec/DWMsCB8DHGw5WQ07Y8v5Bk3yDdsCIJUb+kaadwchQmu5q4e4G90f7+e9wC4n2q27YTbXNduQda4mBlWVw3uCiTbhZZU05OP3wdMf9AOms4uq1nqvFsRBNBm1LmEqqZCXPBn8gq9LW/42YlErQbMONY7Gq3cYt\n-----END CERTIFICATE REQUEST-----\n","format": "pem_bundle","ttl": "43800h"}

The Ansible replace above added the \\n which were needed.

Running this works

curl --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data @/root/payload-csr-cert.json $VAULT_ADDR/v1/pki/root/sign-intermediate --insecure

No matter what combination of user with src: </file/name/here.json> or body: I kept getting 400 errors, which I think are because the spacing in the file

The final bit

Because I’m running Jenkins, i ended up running a stage which runs a simple bash script

curl --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data @/root/payload-csr-cert.json $VAULT_ADDR/v1/pki/root/sign-intermediate --insecure | jq '.data | { certificate }' > /root/intermediatecert.pem

Which outputs (this too has CR’s at the end of each line, i removed them to make the output format better.)

{  "certificate": "-----BEGIN CERTIFICATE-----\nMIIDrjCCApagAwIBAgIUAwj4T6jnamhqZvY/Myv+glS6QZowDQYJKoZIhvcNAQEL\nBQAwGTEXMBUGA1UEAxMOaG9tZXNlcnZlci5sYW4wHhcNMjEwNjAxMTQxNDExWhcN\nMjYwNTMxMTQxNDQxWjAwMS4wLAYDVQQDEyVob21lc2VydmVyLmxhbiBJbnRlcm1l\nZGlhdGUgQXV0aG9yaXR5MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA\n2LRAQvSHDiOV/6IKjc578Ac/eH813Tpwtsn4NgJamastNFPVOn976TtLxyneR43m\nPiknPDncijwmhQNUkXm+00N80sG3hwyfqQe/VyliP7yY86Dxl3GNVpjPeQppUgfc\nzKdrry0PsUnB6S+zTzVJZD6zp2iDSNd5+KnPH7RXZoK7s975HNjF8gz1dep7wLj/\nkShG+muimLDR7BEL1BJ8P0JueROzVXMDHireift1vhG+/Zk+hc51xh/4DJgZvbf0\nLfCnwJN5ABP7BwQttR6/VpGlcXnlkKvAjH6RC8w1RAMgZYOxYNRcUguAgpz4/5YZ\naOKoHU4aDS64IzY1oZP/XQIDAQABo4HWMIHTMA4GA1UdDwEB/wQEAwIBBjAPBgNV\nHRMBAf8EBTADAQH/MB0GA1UdDgQWBBSo2uBJ1wVIYAgcMLYBvgHF89e7AjAfBgNV\nHSMEGDAWgBS6zDHZbTtdDT0LPP6Gkwl0Tg1e9DA8BggrBgEFBQcBAQQwMC4wLAYI\nKwYBBQUHMAKGIGh0dHBzOi8vMTI3LjAuMC4xOjgyMDAvdjEvcGtpL2NhMDIGA1Ud\nHwQrMCkwJ6AloCOGIWh0dHBzOi8vMTI3LjAuMC4xOjgyMDAvdjEvcGtpL2NybDAN\nBgkqhkiG9w0BAQsFAAOCAQEAplFWmfX1L/gIn4N4myzKvcnV48AMhTDj3sxIXXSw\nPS92Ngk67ZxAiTxT2GhdOTcbI8Kh5GyJOmTumMvXntLj3bgjzEqHc4de23113Bf/\nSiT1FdG4Jdq7u/gVpSqelUU8+B+qsVgikvD3UiNhDCHMP13NUJArpxX35fXrLoi1\nMzjCVN+7XRh0LHbtyA+ZlrNwImHgCNcRoap9zpEMw6yYxKmZ0DAYSar2Egd1iS1i\nD4KO+UUxGKoZpA1Yv1EBo/TcG8+F3Bf9O6qjrKDcBm/TFhUDKlYGJPhdmuFN2Sx+\n1MjAhr+NPIe7qdd7eYUgMrTQ4hqk44d2Nt32EatSZqfUMA==\n-----END CERTIFICATE-----"}

Why not run this as shell: or command in Ansible? Simple, special characters are a pig and lifes to short to find the right escaping to get past a colon or a doublequote.

A final set of Ansible is run

    - name: (PKI Server) Submit Signed Intermediate Certificate to Vault       uri:        body_format: json        validate_certs: false        headers:          X-Vault-Token: "{{ vault_token }}"        url: "{{ vault_url }}/v1/pki_int/intermediate/set-signed"        method: POST        src: "/root/intermediatecert.pem"        remote_src: yes        status_code: "204"

Which silently imports the intermediate certificate into Vault

Create a Vault Role to pull the endpoint certs down

Create a payload file

payload-role.json

  - name: Creating a file with content    copy:      dest: "/your path"      content: |        {          "allowed_domains": "homeserver.com",          "allow_subdomains": true,          "max_ttl": "720h"        }

Run this

    - name: (PKI Server) Create Endpoint Role      uri:        body_format: json        validate_certs: false        headers:          X-Vault-Token: "{{ vault_token }}"        url: "{{ vault_url }}/v1/pki_int/roles/homeserver-dot-com"        method: POST        src: "/root/payload-role.json"        remote_src: yes        status_code: "204"

Note the URL

url: "{{ vault_url }}/v1/pki_int/roles/homeserver-dot-com"

This can be changed and will be used by the Ansible role to pull the endpoint certs down.

At this point, the CA Server with a Root and Intermediate CA has been set up.

An Ansible Role

To consume the server an Ansible role is needed to run the following

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \       --request POST \       --data '{"common_name": "test.example.com", "ttl": "24h"}' \       $VAULT_ADDR/v1/pki_int/issue/example-dot-com | jq

And extract the JSON formatted output as a list

{  "request_id": "7544fb94-49aa-8a11-1396-67938b69c3f2",  "lease_id": "",  "renewable": false,  "lease_duration": 0,  "data": {    "ca_chain": [      "-----BEGIN CERTIFICATE-----\nMIIDqDCCApCgAwIBAgIUfg7l07KVEnht4ETStjHLAA3xvp4wDQYJKoZIhvcNAQEL\nBQAwFjEUMBIGA1UEAxMLZXhhbXBsZS5jb20wHhcNMjEwNDE5MTYzNTM1WhcNMjYw\nNDE4MTYzNjA1WjAtMSswKQYDVQQDEyJleGFtcGxlLmNvbSBJbnRlcm1lZGlhdGUg\nQXV0aG9yaXR5MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEApranvA5f\ncP+OySUE+uKNxzR3IkWI59Z0mMYlHYBtAY47V8avXcmwsaFDJiGSq6OXJs8SXKiF\nrXONaQSudWxTO28U8AovcwoV3/C5KsldqbhLMOyor+XDWYQP7GVXGuKYhn9C+kaF\nbiyaPE0SsAb6Yl3emVl1UCq9kBbQm1XmFyih7rdREB4XJZW2Mtp5KdzlHf3zGLfB\nEcK/BF0CsC1kI5UkkJVWxLB3hKx2pr3Bl3olufQDDdweKqlg7YtiPPnmxLvuEq5n\nG5h18++mbxywIvU7iehGYDj+EBFV1zrsOmZcqfSC34xRbJLVg8wFhuQ59Zy/k+mh\nEUo63NOrT1lzTQIDAQABo4HWMIHTMA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8E\nBTADAQH/MB0GA1UdDgQWBBQoQnqjxjJ8ExU3SDHRG9M67KNlLDAfBgNVHSMEGDAW\ngBSnG76ILbfk2VdLrRd6m6bVYK3qRjA8BggrBgEFBQcBAQQwMC4wLAYIKwYBBQUH\nMAKGIGh0dHBzOi8vMTI3LjAuMC4xOjgyMDAvdjEvcGtpL2NhMDIGA1UdHwQrMCkw\nJ6AloCOGIWh0dHBzOi8vMTI3LjAuMC4xOjgyMDAvdjEvcGtpL2NybDANBgkqhkiG\n9w0BAQsFAAOCAQEAegVUIb5SFR/bXo33xqjuocINJLfec22dgjgBO0lMOGp32GFB\nIp4clgfNU1Dw8yGQEoRQHEfWDjZeGdPuCYbgu/ucLW1pLrBiQqO/XgVeFQhwnJp6\nza0NLGnmaDd5k8z1+VJ79tJa0Wb2WRxWxYqu6lsqtYPAj9jD4DM73tfGR76/rCiK\nRB9DRwlr9UTm9h0cw84UpPwaX33kfiIKBeyY9/hUe3pBzGuNGuDgjOiaMph0v+9z\nwiyNCzcR0AN0y1qDVvsqABH9GJ0BpaZ48GbZTZm91Q/kg+HdNlPkgwqpY8+V8aMa\n1CyvvvJeega8q9Evg1v68MajkFFT37YI5XESCw==\n-----END CERTIFICATE-----"    ],    "certificate": "-----BEGIN CERTIFICATE-----\nMIIDZjCCAk6gAwIBAgIUPwOpEOAo39zYqPlQe83UjAH1R14wDQYJKoZIhvcNAQEL\nBQAwLTErMCkGA1UEAxMiZXhhbXBsZS5jb20gSW50ZXJtZWRpYXRlIEF1dGhvcml0\neTAeFw0yMTA0MTkxNjM5MjVaFw0yMTA0MjAxNjM5NTVaMBsxGTAXBgNVBAMTEHRl\nc3QuZXhhbXBsZS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDP\nfe4nVV+AFnRUgW33qXyCOC4y7ruzSm9P6SiEirTFT+MmNtHEGPBGImGFBLYrSqMo\nePiO3GolpHWezY73cvRzFWM/ZLTmfjqsFNsQmP4tS1bbKa7Bn2ZcypbJNENr+/d1\nBkv9mIJuJwI1sQLI2yQe8KZT3HLrtr4tbEGXnY/tZD3rm+kgZNvCq0GpTARATCkZ\n3qUgOT10O5dMYe8Y2GGCZisiviKeGP1S42EL15QsoLhvIui2sZws7sa6LLeWMNYM\nIsWRAYsTkDffFRWEx/yK1axr0Heo9TrRVBuNfRkU8X/p1Ls3kXUB/HMF9RsW53uZ\nDPJbXOHDSJVhkubPoDxXAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgOoMB0GA1Ud\nJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAdBgNVHQ4EFgQU0UKGmwpRpQsi+wTo\nviMS6w6iEkowHwYDVR0jBBgwFoAUKEJ6o8YyfBMVN0gx0RvTOuyjZSwwGwYDVR0R\nBBQwEoIQdGVzdC5leGFtcGxlLmNvbTANBgkqhkiG9w0BAQsFAAOCAQEAhRcjJf6T\nve9hrVsYSVZLGEOmfYoVJ9J2u7fxOfIMnBYJ0EEm047my6Qj5wmu00HSefhheBK1\n46XK5iBlpdSVrV28IjE8PDnWotWVyYhy9vq+LdVbrhHV3JWbRtM21AXkL/73dB3h\nBC3o+lRjmw+ySSEFlfugP+a9ULqwWtuQH+uFWq86F2Ar4y5Sl4TnIiZ9i22Yx+i4\nKcwPRPFsEyCvwRh1Ih/r/UTsrF0lqE7iQ77Yo9I9MuNKMc6y3+fI3vZWTED0QIfz\nVlb+A5Q9AfGhfaygc2fzJ9w/gnZ+D6wp/mQWQz2xNxs3iaDQf8xHpXywYNGbkVXX\ny/cTj/D8a8iAJA==\n-----END CERTIFICATE-----",    "expiration": 1618936795,    "issuing_ca": "-----BEGIN CERTIFICATE-----\nMIIDqDCCApCgAwIBAgIUfg7l07KVEnht4ETStjHLAA3xvp4wDQYJKoZIhvcNAQEL\nBQAwFjEUMBIGA1UEAxMLZXhhbXBsZS5jb20wHhcNMjEwNDE5MTYzNTM1WhcNMjYw\nNDE4MTYzNjA1WjAtMSswKQYDVQQDEyJleGFtcGxlLmNvbSBJbnRlcm1lZGlhdGUg\nQXV0aG9yaXR5MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEApranvA5f\ncP+OySUE+uKNxzR3IkWI59Z0mMYlHYBtAY47V8avXcmwsaFDJiGSq6OXJs8SXKiF\nrXONaQSudWxTO28U8AovcwoV3/C5KsldqbhLMOyor+XDWYQP7GVXGuKYhn9C+kaF\nbiyaPE0SsAb6Yl3emVl1UCq9kBbQm1XmFyih7rdREB4XJZW2Mtp5KdzlHf3zGLfB\nEcK/BF0CsC1kI5UkkJVWxLB3hKx2pr3Bl3olufQDDdweKqlg7YtiPPnmxLvuEq5n\nG5h18++mbxywIvU7iehGYDj+EBFV1zrsOmZcqfSC34xRbJLVg8wFhuQ59Zy/k+mh\nEUo63NOrT1lzTQIDAQABo4HWMIHTMA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8E\nBTADAQH/MB0GA1UdDgQWBBQoQnqjxjJ8ExU3SDHRG9M67KNlLDAfBgNVHSMEGDAW\ngBSnG76ILbfk2VdLrRd6m6bVYK3qRjA8BggrBgEFBQcBAQQwMC4wLAYIKwYBBQUH\nMAKGIGh0dHBzOi8vMTI3LjAuMC4xOjgyMDAvdjEvcGtpL2NhMDIGA1UdHwQrMCkw\nJ6AloCOGIWh0dHBzOi8vMTI3LjAuMC4xOjgyMDAvdjEvcGtpL2NybDANBgkqhkiG\n9w0BAQsFAAOCAQEAegVUIb5SFR/bXo33xqjuocINJLfec22dgjgBO0lMOGp32GFB\nIp4clgfNU1Dw8yGQEoRQHEfWDjZeGdPuCYbgu/ucLW1pLrBiQqO/XgVeFQhwnJp6\nza0NLGnmaDd5k8z1+VJ79tJa0Wb2WRxWxYqu6lsqtYPAj9jD4DM73tfGR76/rCiK\nRB9DRwlr9UTm9h0cw84UpPwaX33kfiIKBeyY9/hUe3pBzGuNGuDgjOiaMph0v+9z\nwiyNCzcR0AN0y1qDVvsqABH9GJ0BpaZ48GbZTZm91Q/kg+HdNlPkgwqpY8+V8aMa\n1CyvvvJeega8q9Evg1v68MajkFFT37YI5XESCw==\n-----END CERTIFICATE-----",    "private_key": "-----BEGIN RSA PRIVATE KEY-----\nMIIEowIBAAKCAQEAz33uJ1VfgBZ0VIFt96l8gjguMu67s0pvT+kohIq0xU/jJjbR\nxBjwRiJhhQS2K0qjKHj4jtxqJaR1ns2O93L0cxVjP2S05n46rBTbEJj+LUtW2ymu\nwZ9mXMqWyTRDa/v3dQZL/ZiCbicCNbECyNskHvCmU9xy67a+LWxBl52P7WQ965vp\nIGTbwqtBqUwEQEwpGd6lIDk9dDuXTGHvGNhhgmYrIr4inhj9UuNhC9eULKC4byLo\ntrGcLO7Guiy3ljDWDCLFkQGLE5A33xUVhMf8itWsa9B3qPU60VQbjX0ZFPF/6dS7\nN5F1AfxzBfUbFud7mQzyW1zhw0iVYZLmz6A8VwIDAQABAoIBADMwNhiuDyliYMCY\nTbDTt0vI4FzgWJ4atutX8g8AySgEVV2QGJ/wJxamVLikOOzlNOs/LNLRvb4bnIjY\n3XRef8AEfr+c8KQMcB0T6BdoJwy1kW/wEJTj5jTuJdTtd9SkDKBqNUUS4tqZ9QmZ\n6b3zki2v4Ni/gfp00uYR1vy4elFt/yjTmA0lhY4onPNPqXNWCF1LY38xNegBQZg4\nrYkSFvIOr7bl7Rk2JljWCxZZef5kEIFGdoqus5KekzW8Li9s2XVVXWEfoY/r/NI1\nHgAwHugwe3+8ve3YKLIIKJqGvYk+8Lp8rXPsf9GRZxpxx/yER4NX8+83DndBnfQO\n7G/BOwECgYEA/0ZAU6h+1cW+PxQ+CuSdbClXhKgqTN4pRwCbrGUQ3y2KGsGUzZW/\nUuO/XIN/hkx0SkKz10TvUV5o4SXh5nY7s+3OCsnc7fBx3Gv4vHCWQ4t9JNxeH6vP\nzNKjQeceVzlniyxKlW9OvVDXZU6nv27LBRH1fIYp9QRbc7yHzUQXLCcCgYEA0BTp\nD3CXELJYoV0/LFvRQqZ+P1Xo26qa9xjRMI+Owr2nNuzqcMl4dBvZWNNCpDwqrKn6\n6iiWXYhwxvvKkJe9QgE3LCvkmluGiBDpK4U2rVbB/LoBAtNgGdnxaP9t9OfhtQC3\nPjhVJvBVbk3mFWG950Ua5E1cgl6n4xtGoHgxHFECgYBaXOTieE/FnoUU0TaRJpIv\nOoc3d0vZ//5+mtGAeho51mX/yKzDBZI/Zk1UE1xuDtxPeUMuHcHVfOUFZiKMMSg7\nLh/0o7ZoJ+g2TaY0FmqqqFL5XGSZM3mQmLOf3Y9Y8wIbOud/9HHcBCTrQKeS1UZa\nmhvbI6bwi8VPt9oeqE7HmwKBgHqOVlbJsbAb2yfvi+3MhowDFAipyOTYrz0qWMuJ\nQkRg/8PR9qNHhrKcVH+ErpOc/GWGGEsibK3aVtJcKwrO1KGzpZNWpuZjUfGCRFNl\nuraNiuQXidDoPon7W7zD9Tdx+/Zn3YXAGCc/FpJJP2MIlplIknY1Om9u4ONahVau\nc/6BAoGBAIfW6PPFjXUUn3LvcswvQAahnclYIW+Ml6peerfMr5kq0TTWykrMN4nc\nLl9oGG2ifigSUo/mz2cfemExO+hgXJP+UOOKi2LkUUpV9x1fzJFHYutiIyV4mBPd\nTFPCpLJjz3T4Nn18nzGmrQiQT6ymgt4KDx6cW+xbOLTuNsNg+0ib\n-----END RSA PRIVATE KEY-----",    "private_key_type": "rsa",    "serial_number": "3f:03:a9:10:e0:28:df:dc:d8:a8:f9:50:7b:cd:d4:8c:01:f5:47:5e"  },  "wrap_info": null,  "warnings": null,  "auth": null}

While I’ve not done this yet, the Ansible task for the role would look something like this to extract the certificate:

- name: (PKI Server) Submit Signed Intermediate Certificate to Vault   uri:    body_format: json    validate_certs: false    headers:      X-Vault-Token: "{{ vault_token }}"    url: "{{ vault_url }}/v1/pki_int/issue/homeserver-dot-com    method: POST    body:      common_name: "{{ servername}}.{{domainname}}"       ttl: "15d"    remote_src: yes    status_code: "204"

This will be output as JSON and using similar filter and lists the certs can be extracted and written out to file for the applications as needed.

Thoughts

This is run once code, it needs work to make it immutable, which can be done quickly using creates: and checks in the code.

I’d love to know why the JSON output ansible creates for the CSR doesn’t work in Ansible.

This is the play failed output

TASK [(PKI Server)Sign Intermediate Certificate with CSR] **********************fatal: [localhost]: FAILED! => changed=false   cache_control: no-store  connection: close  content: |-    {"errors":["error parsing JSON"]}  content_length: '34'  content_type: application/json  date: Fri, 28 May 2021 14:38:30 GMT  elapsed: 0  json:    errors:    - error parsing JSON  msg: 'Status code was 400 and not [200]: HTTP Error 400: Bad Request'  redirected: false  status: 400  url: https://127.0.0.1:8200/v1/pki/root/sign-intermediate

Using Jenkins it would be quicker to run it as a set of Bash scripts to this point.

Tech Blog