Ubicloud - Increased provisioning times for GitHub runners – Incident details

Increased provisioning times for GitHub runners

ResolvedDegraded performance1 hour 56 minutes

Affected

Ubicloud Runners

Degraded performance from 5:20 PM to 7:17 PM

  • UpdateUTC

    We investigated the cache slowness. On October 29, Cloudflare had issues on several of their services. From approximately 09:00 to 22:00 UTC, Cloudflare sent our cache traffic to its Prague location instead of Frankfurt, and Cloudflare R2 had increased latency. Because of this, cache upload and download speeds were much lower than normal.

    Cache performance has been normal since approximately 23:00 UTC on October 29.
    We are working with Cloudflare to find the root cause, and to prevent and detect similar problems earlier in the future.

  • ResolvedUTC

    Wait times for GitHub Actions runners are back to normal.

    Cause: Today we saw a surge in demand that put pressure on our capacity at peak times, especially while EU evenings and US mornings overlapped. This week the overlap is an hour longer than usual because Europe has already switched to winter time and the US hasn't yet. Slowness in Cloudflare R2, which we use for cache storage, also made jobs run longer. This further added to queue times.

    We're adding capacity to handle a continuous increase in overall demand. We apologize for the delay in processing times.

  • MonitoringUTC

    Wait times are improving. We're processing the remaining backlog and monitoring until queue times are back to normal.

  • InvestigatingUTC

    We're seeing longer than usual wait times for GitHub Actions runners to start, due to high demand. We're investigating and will post updates here.