Ubicloud - Degraded git checkouts on GitHub Actions runners (upstream GitHub) – Incident details

Degraded git checkouts on GitHub Actions runners (upstream GitHub)

Resolved
Degraded performance
Started 4 months agoLasted 175 hours 20 minutes

Affected

GitHub Actions

Degraded performance from 11:02 AM to 6:57 PM

Updates
  • Resolved
    UTC
    Resolved

    GitHub confirmed that they resolved the issue on their edge servers.

  • Update
    UTC
    Update

    We are continuing to work on a fix for this issue.

  • Update
    UTC
    Update

    We are monitoring the issue with GitHub's servers.

  • Identified
    UTC
    Identified

    We have traced this to an upstream GitHub issue affecting git data transfer. Connections to GitHub establish normally (DNS, TCP, and TLS are healthy), but transfers stall mid-stream, producing very slow clone and fetch speeds, HTTP/2 stream cancellations, or checkout steps that hang for several minutes. This is occurring across multiple regions and is not specific to Ubicloud infrastructure. The behavior is being reported and tracked by others in the GitHub community here: https://github.com/orgs/community/discussions/196638. We will continue to monitor and update as GitHub addresses the underlying problem. If you need to mitigate in the meantime, the following can help on affected jobs: shallow clones (fetch-depth: 1), forcing HTTP/1.1 (git config --global http.version HTTP/1.1), and adding retry logic around the checkout step.

  • Investigating
    UTC
    Investigating

    We are investigating reports of slow or failing git checkout and git fetch steps on Ubicloud GitHub Actions runners. Jobs may stall, run very slowly, or time out during the checkout step. We are working to confirm the source.