Introduction
This article is part 2 of my series on GitHub self-hosted runners. In part 1 I created a Hyper-V virtual machine, installed Ubuntu Server 20.04, and set up the GitHub runner.
I then tested the runner by creating a new GitHub workflow that runs “Hello World”. Notice the “workflow_displatch” event. This supports manually running workflows.
|
1 2 3 4 5 6 7 8 9 |
name: Hello-World on: [push, workflow_dispatch] jobs: build: # The type of runner that the job will run on runs-on: self-hosted steps: - name: Say Hello World run: echo Hello, world! |
At the virtual machine, the following output is displayed when running the workflow:
|
1 2 3 4 5 6 7 8 |
$ ./run.sh √ Connected to GitHub Current runner version: '2.285.1' 2022-01-02 23:19:22Z: Listening for Jobs 2022-01-02 23:31:37Z: Running job: build 2022-01-02 23:31:43Z: Job build completed with result: Succeeded |
The output from the workflow is stored in GitHub and can be viewed in the GUI and the logs.
Now that I know that you can run a workflow in the self-hosted runner, what can we accomplish via runners? Let’s try patching Ubuntu.
Create a new GitHub Workflow and paste the following code. Name the file “.github/workflows/01-update-ubuntu.yml”. The reason for this name will be clear later in this article. Also note “workflow_call” in the “on” event.
|
1 2 3 4 5 6 7 8 9 10 11 |
name: Update-Ubuntu on: [workflow_dispatch, workflow_call] jobs: update-os: # The type of runner that the job will run on runs-on: self-hosted steps: - name: Update Ubuntu run: | sudo apt-get update -y sudo apt-get upgrade -y |
In the GitHub GUI, go to Actions. Click on the workflow that you just created. Click on the button “Run workflow”. In the popup dialog click “Run workflow”. Notice that the run is “in progress” and does not complete”.
In the ssh session connected to the virtual machine, notice that the runner is displaying:
|
1 2 3 4 5 6 7 8 9 10 |
$ ./run.sh √ Connected to GitHub Current runner version: '2.285.1' 2022-01-04 23:01:25Z: Listening for Jobs 2022-01-04 23:01:56Z: Running job: build 2022-01-04 23:02:01Z: Job build completed with result: Succeeded 2022-01-04 23:02:19Z: Running job: update-os [sudo] password for jhanley: |
The running is prompting for the user’s password in order to run sudo. Enter the password. The runner now completes successfully after a minute or two.
Sudo Password Problem
The issue with the previous workflow pausing to receive password input is caused by sudo requiring the user to enter its password. This can be solved by removing the requirement for a password for this user to run sudo.
In the Ubuntu terminal session, start visudo:
|
1 2 |
sudo visudo [sudo] password for jhanley: |
Add the following line. Replace jhanley with your username:
|
1 |
jhanley ALL=(ALL) NOPASSWD:ALL |
Run the workflow again and verify that a password is not required.
This brings up an item to consider. I am running the GitHub runner as my user identity “jhanley”. Next time, I would create a new user named something like “ghrunner”. I have not yet configured the running as a Linux service, but the identity will probably be important for that step as well.
Installing Packages
I want my runner to be able to build, test and deploy applications written in .NET, Python and PHP as well as build applications in containers. Installing applications and packages can be tedious. Since we can create and run GitHub workflows on the runner, we can put these scripts and their execution under source code control.
If you create separate GitHub workflows for each application or package, you will quickly have a directory full of workflow files. Creating a naming convention to simplify what type of workflow but also the order that the workflows should run which configuring a new running.
I start the workflow file name with “0” to indicate runner setup workflows. For example “01-update_ubuntu.yml”, 02-docker-install.yml”, etc.
Installing Docker
Create a new GitHub Workflow and paste the following code. Name the file “.github/workflows/02-docker-install.yml”. The reason for this name will be clear later in this article. Also, note “workflow_call” in the “on” event.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
name: Docker-Install on: [workflow_dispatch, workflow_call] jobs: install: # The type of runner that the job will run on runs-on: self-hosted steps: - name: Install or Update Docker run: | if [ -x "$(command -v docker)" ]; then echo "Update docker" sudo apt update -y sudo apt-get --only-upgrade install docker-ce -y else echo "Install docker" sudo apt update sudo apt upgrade -y sudo apt-get install apt-transport-https -y curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable" sudo apt update apt-cache policy docker-ce sudo apt-get install docker-ce -y sudo usermod -aG docker jhanley echo "User jhanley added to docker group. Re-login to assume docker group membership." fi |
Notes:
- The username is hardcoded in the workflow. This should be moved to a variable.
The Docker install workflow first checks if Docker is already installed, and if true updates it instead of trying to install again:
|
1 2 3 4 |
if [ -x "$(command -v docker)" ]; then echo "Update docker" sudo apt update -y sudo apt-get --only-upgrade install docker-ce -y |
If Docker is not installed, then it is installed:
|
1 2 3 4 5 6 7 8 9 10 |
else echo "Install docker" sudo apt update sudo apt upgrade -y sudo apt-get install apt-transport-https -y curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable" sudo apt update apt-cache policy docker-ce sudo apt-get install docker-ce -y |
The user “jhanley” is added to the sudo group so that password prompts are not required to run Docker commands:
|
1 2 3 |
sudo usermod -aG docker jhanley echo "User jhanley added to docker group. Re-login to assume docker group membership." fi |
Calling Workflows from another Workflow
GitHub supports calling one workflow from another. You can create one workflow that then calls a series of workflows. This could be used to set up a GitHub self-hosted runner with the required packages for a particular type of runner. For example, a runner setup with Docker and PHP 8.0, another runner setup with .NET Core 5.
For a workflow to be reusable, the values for the “on” event must include workflow_call.
Let’s create a workflow that calls the previous two workflows. This will update Ubuntu and install Docker. This can then be extended to call additional workflows as we build our library of workflows.
Create a new GitHub Workflow and paste the following code. Name the file “.github/workflows/00-setup-runner.yml”.
|
1 2 3 4 5 6 7 |
name: 00-Setup-Runner on: [workflow_dispatch] jobs: call-update-ubuntu: uses: hanley-dev/self-hosted-runner-ubuntu/.github/workflows/01-update-ubuntu.yml@main call-install-docker: uses: hanley-dev/self-hosted-runner-ubuntu/.github/workflows/02-docker-install.yml@main |
Notes:
- I am not sure why GitHub requires the full path to the workflow which includes the repository owner and repository name. When calling one workflow from another workflow, it would make sense to be able to just specify the workflow file name.
- The workflow has hardcode names. These should also be moved to variables.
More Information
- Using self-hosted runners in a workflow
- Events that trigger workflows
- Manually running a workflow
- Reusing workflows
Summary
This was an interesting exercise. The power of GitHub Workflows and Self-Host Runners is very interesting. There are many tools to automate building systems, such as Terraform which is now of my favorites. I can imagine GitHub extending its workflows and actions to step on the features by many third-party tools. Imagine if Microsoft bought HashiCorp and merged Terraform into the GitHub family. Would the market accept that monopoly? Would the two platforms be better together or separate?
Once I have more experience with self-hosted runners and have written more runner package installation workflows, I will publish them as a GitHub repository.
Photography Credit
I write free articles about technology. Recently, I learned about Pexels.com which provides free images. The image in this article is courtesy of Magda Ehlers at Pexels.
I design software for enterprise-class systems and data centers. My background is 30+ years in storage (SCSI, FC, iSCSI, disk arrays, imaging) virtualization. 20+ years in identity, security, and forensics.
For the past 14+ years, I have been working in the cloud (AWS, Azure, Google, Alibaba, IBM, Oracle) designing hybrid and multi-cloud software solutions. I am an MVP/GDE with several.
2 Pingbacks