On my homelab, I’ve got a bunch of tasks which I need to run automatically on some of the servers. Checking certs are up to date, moving files around, clearing logs down.

Historically the way to do this is as a CRON job, these work, however, don’t scale well, are difficult to centrally manage and unless you write in a bunch of error checking how do you know if it worked?

Over the years I’ve migrated from CRON to Jenkins, this worked well for me, and despite the hate, Jenkins gets, for what I’ve been using it for it worked well.

Then I moved to Rundeck an excellent product, with at the time awful onboarding with a terribly documented onboarding process (search this blog if you want to know more and how to get Rundeck On Prem setup)

The current tool of choice for running pipelines on a schedule which I’d like to get working are Pipelines using Gitlab CI/CD

All my code is in gitlab.com and using the pipeline tooling within it seems to be the obvious thing to do to have one less thing to manage or run (not Rundeck or no Jenkins)

It seems from my reading that the GitLab pipelines support bash, running things on remote boxes using gitlab-runners so seems to be the place to head to..

Disclaimer

I’m not a developer, I am at best a system administrator. (some who work with me might disagree) however, while I manage people day to do I run as much tech at home as I can so I understand the technologies when I’m having a conversation about it.

This is my journey, I had a set of things I wanted to do (below) and this is how I solved the problem, if there is a better way, a more efficient way please reach out on Reddit (mightywomble)

What I was finding by searching

To get to this point, yet again as I find time and time again when I try and get my head around setting up DevOps style tools. There is lots of documentation however I found it all over the place, very focused with a lot of posts and youtube videos obviously spawned off specific tasks. Unfortunately, they were usually based around Docker and I wasn’t looking to use Docker.

What Do I want to do?

So this is what I’m looking to do with my pipeline

  • Setup a GitLab runner on a remote server
  • Have the GitLab runner use shell (locally) to execute code
  • Have the code in a Gitlab.com project run on that remote server as a bash command
  • Run that code at 8 am every morning
  • Know if the code has run or not.

Connecting Remote Servers to GitLab (SaaS)

GitLab Runners

So the first question is “What is a Gitlab runner?”

Put quite simply it’s an agent that runs on a remote server and talks back to the SaaS version of Gitlab (There’s a different method on the self-hosted version) and dictates what method of execution (shell, ssh, docker container etc) can be run on that remote server

Runners come in multiple flavours, shared, locked and associated with a project and its the latter I’ll be using moving forward.

Runners can run on Windows, OSX or Linux servers.

Choose your Project

I’m assuming you have a project setup in Gitlab and some code to un under that project

Within that project view installation instructions in GitLab by going to Settings > CI / CD, expanding the Runners section, and clicking Show runner installation instructions.

Then in the top right click on Show runner installation and registration instructions

This will display a popup where you can choose the Server type and architecture which I’m installing on Ubuntu 22.04 servers
For each operating system, there will be displayed the installation instructions
# Download the binary for your system
sudo curl -L --output /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64

# Give it permission to execute
sudo chmod +x /usr/local/bin/gitlab-runner

# Create a GitLab Runner user
sudo useradd --comment 'GitLab Runner' --create-home gitlab-runner --shell /bin/bash

# Install and run as a service
sudo gitlab-runner install --user=gitlab-runner --working-directory=/home/gitlab-runner
sudo gitlab-runner start

Once all is installed it’s time to register the runner against your project.

sudo gitlab-runner register --url https://gitlab.com/ --registration-token GR1348941d3yvuDJDJMEKEIMDUwcLk

Heading back to the Gitlab.com interface the runner(s) you’ve installed will be listed

Important things to remember when installing the runners

  • During the installation of the runners you’ll be asked to add tags, these are important as later on, we are going to link your scheduled pipeline to the tags.
  • Installing the gitlab runner creates a user gitlab-runner on the remote server which will be where (unsurprisingly) gitlab will run things.
  • In /home/gitlab-runner/ on the remote server remove .bash_logout if you don’t you’ll get an error like the following when you first run the pipeline code
The Runner of type Shell don't work: Job failed (system failure): preparing environment:

What have we done?

At this point we have

  • Setup a project in GitLab SaaS
  • Assign Gitlab Runners to the project
  • Setup tags and method of running.

The next question I had was how do I make the GitLab repository a pipeline to run things?

Setup a GitLab Pipeline

The .gitlab-ci.yml file

The pipeline is controlled by a file called .gitlab-ci.yml which is a YAML file that contains blocks with instructions to be run on the remote Gitlab Runner server as the user gitlab-runner

In your project repo you want to run the pipeline in creating an empty file called .gitlab-ci.yml (yes there is a dot at the front of that, it makes it a hidden file) this can be done either in a cloned copy of the project repo or directly in the GitLab web editor which if you use there are lots of example templates.

The shell template for example looks as follows

# This file is a template, and might need editing before it works on your project.
# To contribute improvements to CI/CD templates, please follow the Development guide at:
# https://docs.gitlab.com/ee/development/cicd/templates.html
# This specific template is located at:
# https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Bash.gitlab-ci.yml

# See https://docs.gitlab.com/ee/ci/yaml/index.html for all available options

# you can delete this line if you're not using Docker
#image: busybox:latest

#before_script:
#  - echo "Before script section"
#  - echo "For example you might run an update here or install a build dependency"
#  - echo "Or perhaps you might print out some debugging details"

#after_script:
#  - echo "After script section"
#  - echo "For example you might do some cleanup here"

#build1:
#  stage: build
#  script:
#    - echo "Do your build here"

Setup .gitlab-ci.yml

For my automation I want the following to happen

  • Download the repo
  • Run ansible code

However to show an example the pipeline runs two blocks a test block and a deployment block. We could run pre and post execution blocks to set things up, build blocks if for example I was running some automation to create a docker image I could have stages to build, test, upload and deploy

my code looks like this

test1:
  stage: test
  tags:
    - plex
  script:
    - pwd
    - whoami
    - ls
    
deploy1:
  stage: deploy
  tags:
    - plex
  script:
    - ansible-playbook movefiles.yaml
  only:
    - schedules

So what’s happening here?

The first test1 block runs on the plex gitlab-runner (remember those tags when we setup gitlab-runner)

It runs 3 bash commands on the plex server

If the test is successful (admittedly it’s not much of a test, it’s just an example) deploy1 is run, if it failed deploy1 won’t run.

The second deploy1 block again runs on the plex server and executes the ansible-playbook code to stop it from running each time I update the code in the GitLab repo I’ve added the only: section to instruct to only run the pipeline on a schedule.

Commit this code.

Head to CI/CD -> Pipeline and click on Run Pipeline

Test

Inevitably the first few runs of these pipelines are going to have problems and you’ll want to troubleshoot and see what is happening on the remote server when the code is run.

To troubleshoot head over to CD/CI -> Jobs and you’ll see what is working and what is not.

Clicking on a job block will display the output of the job

Schedule

At this point, you should have a working pipeline and you’ll want to run it on a schedule, this is done by heading to CI/CD -> Schedules and clicking on New Schedule

This will display a pretty simple form which is based on the CRON syntax for times, as we are working out of master leave the branch the same and click on Save Pipeline Schedule.

This will display the pipeline

And you’re done.

A mail will be sent out to the email associated with the GitLab user account you are using on success or fail.

Next steps

So this is very high level, designed to get me up and running. Ideally, there is more to be done here

  • Run this all as code
  • Add a better testing block
  • Understand the runner executors more
  • What other options are available in gitlab-ci.yml file

Thoughts

Once I got my head around this and solved a few issues it’s actually pretty powerful as a solution, I do like that I don’t need to host it, it runs off YAML and will run anything which can be run in bash.

Further Information



By davidfield

Tech Entusiast