GitHub Actions Concurrency Group Trap

Idea

As a developer, I've been exploring GitHub Actions as the primary automation tool for a new project. For the first month, I focused on fulfilling 80% of our team's automation needs. As the project grew, we incrementally improved the workflows and fixed edge cases.

Initial Implementation

Initially, we used the concurrency feature in many workflows. However, we encountered problems with only one where it started troubling issues. The workflow we were having issues with was run on main branch merges.

Our workflow was configured with the concurrency feature turned on:

concurrency:
  group: ci-master
  cancel-in-progress: true

We prevented running jobs in parallel for the given group, and when a new job was added to the queue, it cancelled the running job.

Better Implementation

At some stage, we realized that we needed to have more control over the deployment script run and prevent it from being cancelled. Cancelled deployment jobs troubled us sometimes bad deployments and broken staging application instances.

The simplest idea was to get rid of the cancel-in-progress feature:

concurrency:
  group: ci-master

Does it really work?

Although Github documentation is usually pretty clear, it seemed that concurrency feature is simple enough and there is not much to read about. However, things became complicated when the queue grew.

At the time of this article's publication (October 2023), the concurrency groups feature in GitHub Actions "silently" canceled some of the jobs from the queue. It seemed that it kept a maximum of one latest workflow/job runs in the queue.

Summary

I've worked with many CI/CD tools, and unexpected cancellations are a bit odd. Although when you're a bit more careful with reading the documentation, you can find the behavior described, but it's not immediately obvious.