Summary

On October 13, 2026, from 16:24 to 19:50 UTC, GitHub Actions jobs on Ubicloud runners took much longer than usual to start. Lock contention in our control-plane database cut runner placement to about half its normal rate, and the backlog was cleared by 21:10 UTC.

Impact

• Job start times: between 16:20 and 21:00 UTC, jobs waited a median of about 4 minutes for a runner (normally 14 seconds), and 1 in 20 waited more than 28 minutes.

• Throughput: from 16:20 to 20:00 UTC we started about half as many runners as on a normal day.

What happened

Our control plane tracks each host's free capacity in one database row per host, and every runner placement locks and then updates the row of the host it picks. On October 13, the runner fleet had high utilization, so concurrent placements often picked the same hosts. At the same time, a sudden burst of additional requested runners triggered an edge case in our allocation algorithm. This edge case more than tripled the rate of capacity checks.

At 16:24 UTC the database tipped into a slow state, most likely in PostgreSQL's row-locking bookkeeping (MultiXacts). Scans of the capacity table slowed from 0.1s to 2s, and slower placements overlapped even more on the same rows, which made the contention worse. The system stayed in that state until we cleared the backlog of waiting runners.

Our admission checks made it worse. They are not atomic: runners that check at the same moment all see the same free capacity and are let through together. Each time capacity seemed to free up, thousands of runners entered placement at once and competed for the little space left, which twice brought the slowdown back after we had cleared it.

We apologize for the impact this event caused our customers.

Timeline (UTC, October 13)

What we're changing

Already merged

• Placements now lock a host before using it and skip hosts that another placement holds, so two placements no longer work on the same database row at once (merged October 19, #6680). In a lab replay of the incident's load, this change removed the slowdown.

• IP addresses and CPUs are reserved without waiting on other placements (October 13-14, #6671, #6683).

• Runners created to catch up on queued jobs now start over a short window instead of all at the same moment (October 16, #6681).

In progress

• Admission will reserve capacity atomically, so a burst can only start as many runners as there is room for. Waiting runners will share one capacity check and be admitted gradually.

• Created a simulator to simulate our runner allocation algorithm to ensure that we don’t run into similar edge cases. Once we confirm the simulator reflects the production environment’s behavior, we will make further updates to our allocation and scheduling logic.

This incident has been resolved. We are going to prepare and provide a retrospective in the upcoming days.

The fix that we implemented didn't have the intended effect due to a high backlog. We took manual action to reduce the backlog. Queue times should be back to normal now. We're monitoring the situation.

We identified the issue's root cause and we implemented a fix. We expect the fix to start improving the queue times in a short amount of time.