From Zero to Code: Using Ansible in Jenkins Pipelines.

From Zero to Code: Using Ansible in Jenkins Pipelines.

Over the last few years, I've run the gambit of CD/CI tools on my homelab and this week has found me go full circle from Jenkins to Rundeck to AWX (Ansible Tower community) back to Jenkins.

Why the change? Twofold really... AWX in its jump from 17 to 18 has stopped using docker and started using Kubernetes which means more underlying infrastructures from what I'm reading right now the "lightweight" versions of K8 install are not playing friendly with AWX. also where I've recently moved to work-wise, are a Cloudbees Partner (Cloudbees have no input into this blog and are not paying me any money or giving me anything to do this, I'm writing this to assist)

Having my own Dev and prod Jenkins to test things out on is always useful.

Disclaimer

  • Spelling should be better than previous posts
  • I'm no guru in this, and will certainly have dialogue around constructive questions and criticism.
  • There is much more to this subject than this.
  • I will always come to these tools as a sysadmin rather than a developer.
  • Having deployed manually as documented here, I wrote the ansible code to redeploy.

The outcome

Let's start at the end, what am I wanting to do?

I have about 10 Ansible playbooks hosted on a self-hosted GitLab-CE which run the backups, updates, data migrations etc on my homelab. These have been running on AWX 17 for about a year.

Once a playbook is run I'd like an alert to end up in Discord

Longer-term, I'll refactor the playbooks into a Jenkins pipeline to give better control especially for the playbook I use to provision machines as this starts with a bash script, then runs an ansible script...  Having Jenkins control this pipeline will also then allow me to run some tests at each stage of the deployment.

The Setup

I run everything on RHEL8 on Proxmox CE on 2 well-specced laptops.

Both Laptops have 16Gb RAM, and the central storage is provided by a centralised 16Tb QNAP NAS mounted as iSCSI storage on the Proxmox OS.

All the monitoring is done by Netdata and all servers are configured by an Ansible Playbook.

Background

Jenkins
Jenkins – an open source automation server which enables developers around the world to reliably build, test, and deploy their software

What is Jenkins?

The best description I've found is

Jenkins offers a simple way to set up a continuous integration or continuous delivery (CI/CD) environment for almost any combination of languages and source code repositories using pipelines, as well as automating other routine development tasks. While Jenkins doesn’t eliminate the need to create scripts for individual steps, it does give you a faster and more robust way to integrate your entire chain of build, test, and deployment tools than you can easily build yourself.
What is Jenkins? The CI server explained
Jenkins offers a simple way to set up a continuous integration and continuous delivery environment for almost any combination of languages and source code repositories

What this breaks down to is, Jenkins is a tool in which you can use your language of choice for automating tasks and takes that code and runs it on remote servers depending on the needs of the job.

Its Java-based (but don't let that put you off)

What is Ansible?

Ansible is a language used for automation, based on python but fed and "controlled" using YAML files, it uses very simple logic to automate all kinds of things from building complex environments to deploying docker containers.

Ansible for me is one of the simpler languages to pick up if you'd like to move forward from a bunch of bash scripts to immutable based code (sets a standard measured each time it's run). There is a learning curve, it's not steep.

Installation

This install will be performed on RHEL8, SELinux is enabled (it doesn't need to be, so I'm leaving it on)

Install Jenkins

Install Java

sudo yum install java-11-openjdk-devel

Install wget

sudo install wget

Check Java is installed

java --version

should return

openjdk 11.0.11 2021-04-20 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.11+9-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.11+9-LTS, mixed mode, sharing)

Download the Jenkins Repo

wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat-stable/jenkins.repo

Import the Repo Key

sudo rpm --import http://pkg.jenkins.io/redhat-stable/jenkins.io.key

Install Jenkins

sudo yum install jenkins

Jenkins will install and start up, it runs on port 8080/tcp which needs to be opened on the firewall

sudo firewall-cmd --add-port=8080/tcp --permanent

Finally, ensure the service starts up each reboot

sudo systemctl restart jenkins
sudo systemctl enable jenkins
sudo systemctl status jenkins

With a fair wind, Jenkins should now be running on

http://<server address or ip>:8080
As a side note, and out of scope for this post, if you want to run on port 80 or using https on port 443, you'll need to put an NginX Reverse proxy in front of the Jenkins server.

Configure Jenkins

Dashboard setup

The first login there is some basic setup that needs to be done to get the server running.

You will be asked to create an admin user (you do not need to use admin as the name)

Finally, Jenkins needs some plugins installed, install the suggested plugins

Wait for this to complete

Once complete the main dashboard will display

Setting your environment

Now Jenkins is installed and set up, the next step is to start getting the tools you'll be using for your workflow connected to Jenkins, what follows are my steps, your journey may take you down your own rabbit holes..

Add GitLab

Requirements for Gitlab – Jenkins Integration

  • Jenkins GitLab Plugin
    Build trigger that allows GitLab to trigger Jenkins builds when code is pushed or a merge request is created.
  • Jenkins Git Plugin
    This plugin allows use of Git as a build SCM, including repository browsers for several providers.
  • Jenkins GitLab Hook plugin
    Enables Gitlab web hooks to be used to trigger SMC polling on Gitlab projects

First, we must create a user that Jenkins will use to interact via the GitLab API. You should set this user as global admin or add as a member to related Group/Project. In order to report build status, developer permission required.

First open GitLab and go to Profile Settings

Then, click “Access Token” and create a Jenkins API Token.

Then, save and copy this API Token. Let’s go on with Configure the Jenkins Server.

Configure the Jenkins Server

Go to Jenkins and install Jenkins GitLab Plugin and Jenkins Git Plugin.

Go to Manage Jenkins -> Configure System and scroll down to the ‘GitLab‘ section. Write a custom connection name, enter the GitLab server URL in the ‘GitLab host URL‘ field and click Add -> Jenkins button.

Then, fill the required fields as shown below with the Jenkins Access API Token which we created in GitLab before.

Then, select this API Token credential and finally click “Test” and see the “Success” message.

Gitlab and Jenkins are now connected.

Setup Ansible in Jenkins

Getting this setup was covered really well in this video

Notifications

Jenkins can send things out to most services and while traditionally this would be mail for which there are some (read: Many) good tutorials on the web. Personally I've found this one to work well

https://www.360logica.com/blog/email-notification-in-jenkins/

I push my notifications to a Private discord channel on my Discord server as its more visually appealing.

Note:
The current version of Jenkins and the discord plugin work in the freeform projects, however are throwing an Embed 0 error in pipelines. I might of tracked this down to the supported fields.

To set this up head to

Dashboard -> Manage Jenkins -> Manage Plugins

Click on Available

http://<server address>/pluginManager/available

Search for Discord (ignore the fact I'm in Installed, you'll need to be in Available)

Click on the Check box and then click on Install without restart at the bottom

Let the Plugin install.

We will cover how to use this below...

Add Nodes

Nodes are remote "servers" (they can be Windows, Mac, Linux devices) which you'll ask Jenkins to run your Pipelines on to get your desired outcome. It is perfectly possible to run Jenkins as its own node, however, if you can, at very least have a small Linux jump box with Ansible installed and /etc/ansible/hosts setup.

If we take the concept of a remote Linux box, it will need

Head to

Dashboard -> Manage Jenkins ->Manage Nodes and Clouds

Click on New Node

Add a new Node with a useful name

Click on OK

The next screen has a lot of questions

This is an overview of the defaults

Jenkins Add
Name Should be the name you entered
Description Enter a descriptio
# of executors 1
Remote root directory /jenkins
Labels build test example
Usage Leave
Launch Method Launch Agents via SSH
Host IP of your remote server
Creds See Post
Host Key Verification Strategy Non-verifying Verification strategy 
Availability Keep online as much as possible

The area which will need to be set up is connecting to the remote node, this is why passwordless SSH is suggested above. If this has been set up then a private key should be available for the user.

We set this up as follows

Enter the IP or DNS of the remote jumpbox

Click on Add then Jenkins

Select under Kind  from the dropdown SSH Username with Private key

  • Enter the username
  • Enter the key directly
  • Paste the Private key into the text box
  • Click on Add

Selec the new credential

This will take you back to the following screen (again, excuse the screenshot, this node was setup the same way as explained above)

Click on Launch Agent and hopefully a lot of text will scroll passed ending with

Agent successfully connected and online

If you get an error, it will usually be

  • Java isn't installed on the remote machine
yum install java-11-openjdk
  • The permissions are not set on /jenkins
  • /jenkins doesn't exist
  • The passwordless SSH isn't working.
  • There are networking issues over port 22/tcp between the Jenkins server and the Node

These are all things you can google to fix.

If the Agent connects, click on Back to List and you should be greeted with the following.

Connect as many nodes across multiple OS as needed... if it runs Java, it should work...

Bringing it all together

At this point, there is a running Jenkins server, a remote node to run ansible from, connectivity to Gitlab on the LAN which holds Ansible playbooks and the ability to do notifications to Discord.

The next step is to set up a Job and there are two main options

  • Freestyle provides a one hit run of a playbook or a script
  • Pipeline provides a more scripted approach to deploying a CD/CI stream.

The same Ansible playbook will be run in both examples..

Create a Job to run from Jenkins

Freestyle

  • Click on
Dashboard -> New Item
  • Give the Item a name
  • Click on Freestyle project
  • Click on OK

The next page has a lot of options on it, and it really is useful taking a look at what each one does

Screen 1

I've selected Restrict where this project and be run and associated a label of the jump box. If you have clustered jump boxes to run things on this can be handy and ensure the job is running in the right place.

Source Code Management

  • Select Git
  • Enter the git string for your project you'd usually use for the git clone command
  • Make sure the correct branch is selected
  • Select the credentials for Gitlab.

Build Triggers

Because I want my Job to run daily, weekly etc I've selected Build periodically

Jenkins uses a similar notation to Cron

Build Environment

I've selected Delete workspace before build starts because why not ensure its the right fresh build each time

Build

  • Click on add and add Ansible Playbook
  • Whatever you called the Ansible plugin when setup will display here
  • Add the name of the playbook YAML file (if it's in the root of whatever git downloads then just the name if not then folder/playbook.yaml format.
  • I've selected to not use an Inventory file as I detail this in the playbook
  • Select the SSH creds needed to get from your jump box to wherever your ansible-playbook runs. Add new creds here if needed.
  • Click on Become if you need to become root in your playbook

Post Build Actions

  • Click on Add
  • Select DiscordSend from the drop-down
  • Enter the Webhook of your Discord Channel
  • There are other options under advanced, however, they may cause the plugin to bork out. so test.

Click on save

You'll be taken to the Project page (again, yes, I've renamed the project.. it was setup as above)

To test if this all works click on Build Now

A new Job will appear in the build history list

Click on the Job and you'll be taken to the job page

Within here Console View will give a real-time view of what is or isn't happening with your job. If it fails this is the first place to start debugging.

If there are problems, click on Back to Project

Click on configure and you can edit the jobs configuration.

The discord notification is also pretty nice

As It is setup to display Green Red or Yellow depending on the build state and I've ticked the box to include the build log in Discord as well.

Pipeline

  • Click on
  • Give the Item a name
  • Click on Pipeline project
  • Click on OK
Dashboard -> New Item

The next page is simpler than the freestyle project.

Again, I will highlight the options I've changed for this build and it's worth seeing what each of them do...

And while I've changed nothing on this screen for this build options like Build Periodically still exist

The pipeline section uses the Jenkins declarative pipeline language which I found really difficult to get my head around when I started using Jenkins.

Thankfully the post I wrote for myself back then is still on medium

Jenkins — Pipeline (Beginners guide)
The following document is designed to explain what Jenkins Pipelining is, and provide examples of a Jenkins pipeline which runs a basic script on a Jenkins slave. I spent a while searching around…

A Pipeline is basically broken into stages and the stages have steps.

The pipeline above looks like this

pipeline {
    agent { label 'build' }
    stages {
        stage('GITLab Checkout') {
            steps {
                git branch: 'master', credentialsId: 'lancert', url: 'git@gitlab.lan:homelan/ansible/movefiles.git'
            }
        }
        stage('Execute Ansible') {
            steps {
                ansiblePlaybook become: true, colorized: true, credentialsId: 'lancert', disableHostKeyChecking: true, installation: 'ansible', playbook: 'movefiles.yaml'
            }
        }
        stage('Notify sucess') {
            steps{
            discordSend(description: "${currentBuild.currentResult}: Job ${env.JOB_NAME} \nBuild: ${env.BUILD_NUMBER} \nMore info at: \n${env.BUILD_URL}", footer: 'No-Code', unstable: true, link: env.BUILD_URL, result: "${currentBuild.currentResult}", title: "${JOB_NAME} << CLICK", webhookURL: 'https://discord.com/api/webhooks/8404094/yeogAMDU5wq6YFr1h1bUAn6A_9XY2O71dFKOt9lUZrAW6OnFEG7y')
                }
        }
    
    }

}

And tells Jenkins there are 4 things we need it to do

1) Where to run this pipeline (any Node with a label of Build)

2) Pull the git code

3) Run the Ansible Playbook movefiles.yaml

4) Send a message to Discord when done (this is not working in pipeline mode, i think one of the options is broken)

Because the Syntax of the plugins may not be immediately obvious there is also a Pipeline Syntax link which when opened (in a new tab) provides an interface for creating the lines needed for the steps section of the pipeline

Once the Options for the plugin selected in the drop-down are entered clicking on Generate Pipeline Syntax will create the line needed to copy and paste into the pipeline step.

Going back to the Job view the same side menu is available and clicking on Build Now will start a build that will appear under build history.

The console output again is in real time and the output is great for troubleshooting.

However, because this build is created using the pipeline syntax a Stage View is now visible showing the three stages we created. If a stage fails it's visible here making troubleshooting more focused and a build can be restarted from a specific stage rather than from the beginning.

This was a quick overview of the two different build types.

Thoughts

This is a quick dirty overview I've written up after a few days of work to get Jenkins running basic ansible Jobs. there is so much more to the tool which is outside the scope of this post such as Jenkinsfiles (where all this GUI stuff is written in code), different views, dashboards...

There will certainly be follow up posts as I reacquaint myself with this product.

References

Jenkins GitLab Integration
In this article, we will integrate Jenkins and GitLab. GitLab includes Git repository management, issue tracking, code review, an IDE, activity streams...
Jenkins CI service | GitLab
Documentation for GitLab Community Edition, GitLab Enterprise Edition, Omnibus GitLab, and GitLab Runner.

https://www.360logica.com/blog/email-notification-in-jenkins/

How To Set Up Passwordless SSH Login {Instructional Guide}
Set up and enable a passwordless SSH login by generating public key authentication. Connect to a remote server without password requirements.
Jenkins — Pipeline (Beginners guide)
The following document is designed to explain what Jenkins Pipelining is, and provide examples of a Jenkins pipeline which runs a basic script on a Jenkins slave. I spent a while searching around…

Share Tweet Send
0 Comments
Loading...
You've successfully subscribed to Tech Blog Posts - David Field
Great! Next, complete checkout for full access to Tech Blog Posts - David Field
Welcome back! You've successfully signed in
Success! Your account is fully activated, you now have access to all content.