github actions and go private modules

I’ve followed up this post with a new article on Continuous Integration with Go and Github Actions.

As the number of repositories within a project increases so does the complexity of the dependencies. As time goes on you begin to want to share code between the repositories. This is where Golangs Module system comes in handy. Creating new private modules is easy but challenges arise when you want to use those modules in your projects. We’ll explore how to configure local development and GitHub Actions to use private Go modules.

Creating a Private Go Module #

Creating a private Go module is pretty simple. Initialising a new module by running go mod init github.com/{organisation/{name} will create a new module. You can then add your code to the module and commit it to your repository. To make the module private you can push it to a private repository on GitHub.

Now that we have our module we’ll want to use it with other Go projects.

Local Development #

For this we’ll have to alter our local development configuration to allow the Go tooling to access the private module.

The first thing we need to configure on our local development environment is the GOPRIVATE environment variable. This variable tells the Go tooling which modules should be considered private and should not be access via proxies or checksum database. We can set this variable to our GitHub organisation GOPRIVATE=github.com/{organisation}. This variable can include multiple addresses as a comma separated list. This will allow the Go tooling to access any private modules within the organisation.

The next thing we need to do is configure the Go tooling to use our GitHub credentials. There are two ways to configure this. The first is to use the .netrc file. This file is used to store credentials for remote hosts. The file is usually located at ~/.netrc.

The file should contain the following:

machine <host>
login <LOGIN>
password <PASSWORD>

Although the file suggests using a password GitHub does not actually support this. Instead we can use a personal access token. You can create a new personal access token by going to Settings > Developer settings > Personal access tokens. Once you have created a new token you can add it to the .netrc file.

The other authentication method is to use SSH keys. This is the method I prefer as it allows me to use my existing SSH keys. To use this method we need to change our git configuration to use SSH instead of HTTPS. We can do this by adding the following to our git configuration:

[url "ssh://git@github.com/"]
    insteadOf = "https://github.com/"

Now with the GOPRIVATE environment variable and our authentication method configured we can use our private modules in our projects locally.

GitHub Actions #

Now that we have our local development environment configured we need to configure GitHub Actions to use our private modules.

Like local development we’ll need to configure the GOPRIVATE environment variable and our authentication method. This can be like local either using Personal Access Tokens or SSH keys. Recently I used the SSH key method to configure GitHub Actions. In this article we will explore how to configure GitHub Actions to use SSH keys.

The first thing we will configure is the GOPRIVATE environment variable. We can do this by adding the following to our workflow:

env:
  GOPRIVATE: github.com/{organisation}

The next step is configuring the SSH Key. To use this we’ll need to add a Deploy SSH key to our private module. We can do this by going to Settings > Deploy keys and adding a new key. We’ll need to give the key a name and add the public key. The public key can be generated by running ssh-keygen -t ed25519 -C "your_email@example.com" -f github-actions. This will generate a new SSH key pair. We can then add the public key as our Deploy Key. The private key will be used in our GitHub Actions workflow.

Adding the private key to our GitHub Actions should be done using GitHub Secrets. We can add the private key to our GitHub Secrets by going to our organisation settings Settings > Organizations then selecting our organisation then to Secrets and variables > Actions. We can then add a new secret with the name GO_MODULE_PRIVATE_KEY and the contents of the generated private key.

Now that we have our private key configured we can add it to our GitHub Actions workflow. We can do this by adding the following step to our workflows jobs:

- name: Add SSH Go Module Private Key
  env:
      SSH_AUTH_SOCK: /tmp/ssh_agent.sock
  run: |
      mkdir -p ~/.ssh
      ssh-keyscan github.com >> ~/.ssh/known_hosts
      ssh-agent -a $SSH_AUTH_SOCK > /dev/null	
      ssh-add - <<< "${{ secrets.GO_MODULE_PRIVATE_KEY }}"
      echo "SSH_AUTH_SOCK=$SSH_AUTH_SOCK" >> $GITHUB_ENV

In this snippet you can see the use of ssh-add to add the private key to the SSH agent and the use of echo to add the SSH agent to the environment. This will allow the Go tooling to use the private key to access the private module.

The next thing we will have to add now is the Git configuration making Git and the Go tooling use SSH instead of HTTPS. We can do this by adding the following step to our workflow:

- name: Setup access for private go modules
  run: |
    git config --global url."ssh://git@github.com/".insteadOf https://github.com/

Now that we have all of the necessary configuration we can use our private module in our GitHub Actions workflows. Here’s a full workflow example:

name: Go Vet & Test
on:
  pull_request:

jobs:
  test:
    name: "Test"
    runs-on: ubuntu-20.04

    steps:
    - uses: actions/checkout@v3

    - name: Set up Go
      uses: actions/setup-go@v4

    - name: Add SSH Go Module Private Key
      env:
        SSH_AUTH_SOCK: /tmp/ssh_agent.sock
      run: |
        mkdir -p ~/.ssh
        ssh-keyscan github.com >> ~/.ssh/known_hosts
        ssh-agent -a $SSH_AUTH_SOCK > /dev/null	
        ssh-add - <<< "${{ secrets.GO_MODULE_PRIVATE_KEY }}"
        echo "SSH_AUTH_SOCK=$SSH_AUTH_SOCK" >> $GITHUB_ENV        

    - name: Setup access for private go modules
      run: |
        git config --global url."ssh://git@github.com/".insteadOf https://github.com/        

    - name: Run tests
      run: go test

Of course you can make your workflow do more than just run tests. You can use this workflow to build your Go application, deploy to a server or even create a Docker image.

Conclusion #

We explored how to configure local development and GitHub Actions to use private Go modules. We looked at how to configure the GOPRIVATE environment variable and how to configure the Go tooling to use our GitHub credentials. We also looked at how to configure GitHub Actions to use the same configuration as our local development environment.