Ubicloud - Notice history

On July 29, 2026, from 14:51 UTC to 15:28 UTC, GitHub Actions experienced elevated REST API request timeouts and errors, failures registering runners, and delayed workflow run starts for customers whose traffic was served by a single infrastructure site. This was caused by an under-provisioned internal Actions service in that site: under increased load its instances ran out of memory and became unresponsive, and because Actions API requests wait synchronously on that service, requests routed through the affected site stalled and timed out. During the incident, approximately 2% of workflows were delayed. Requests served by other sites remained unaffected. Both standard and larger hosted runners routed through the affected site could see delayed job starts.

The issue was mitigated by scaling out the runner-administration service in the affected site and increasing the replica count, which restored API availability and returned workflow run starts to normal. We are working to add horizontal autoscaling, memory-saturation alerting, and scaling-forecast monitoring for this service, along with responder playbooks, to reduce the likelihood of similar issues in the future.

The degradation affecting Actions has been mitigated. We are monitoring to ensure stability.

We are investigating an issue affecting GitHub Actions. Some customers may experience timeouts or failures with runner registration and workflow runs may be delayed during startup. Our team is actively working to mitigate the impact by scaling capacity across additional infrastructure.

On July 26, 2026 at 21:34 UTC we began seeing intermittent errors on the GitHub GraphQL API. A subset of GraphQL API requests returned HTTP 502 errors in short bursts. During the impact window an average of 0.09% of GraphQL API requests in the affected region failed, with a peak of 0.50% of requests failing during the worst two-minute period at 03:02 UTC on July 27. Requests that failed generally succeeded when retried, and no data was lost or altered. Other GitHub services were not affected.

The errors were traced to a single group of servers handling a share of GraphQL API traffic. Application processes on that group intermittently closed connections before completing responses. Impact ended at 03:52 UTC on July 27 when those processes were replaced, and we resolved the incident at 04:09 UTC on July 27 after confirming error rates had returned to normal.

We are still investigating why those processes closed connections, and that work is being carried out by the team that owns the underlying compute platform. In the meantime we are adding detection and automated mitigation for when a single group of servers behaves differently from its peers.

The degradation affecting API Requests has been mitigated. We are monitoring to ensure stability.

The runners should be back to normal. The upstream github actions API incident has been resolved.

Number of errors from the upstream Github API has reduced. Although, the upstream incident is still active https://www.githubstatus.com/incidents/pz7g535gbs6p

We are continuing to investigate this issue.

On July 25, 2026, GitHub Actions experienced two related periods of degradation that caused some workflow runs to be delayed by more than 5 minutes or end with infrastructure failures.

First period (08:45 – 09:13 UTC): During planned maintenance on a critical-path Redis cluster for Actions, one participating region was left in a degraded state. Separately, an independent capacity operation temporarily removed another region from the cluster and redirected its traffic to the degraded region. This created cross-region inconsistencies in job-assignment state, causing workflow runs to be delayed, exhaust retries, or fail outright. At peak, about 7% of runs were delayed by more than 5 minutes, and 25% of runs failed with an infrastructure error during the course of the incident. We mitigated the incident at 09:13 UTC by returning traffic to its normal distribution.

Second period (12:08 – 12:48 UTC): As part of mitigating the first incident, traffic was returned to the regional instance that was still undergoing its capacity increase. Multiple Redis nodes in the scaling region experienced failures, increasing traffic to healthy nodes and causing connection limits to be reached on many nodes. At peak, 30% of runs were delayed by more than 5 minutes, and 60% of runs failed with an infrastructure error during the course of the incident. We mitigated the incident at 12:48 UTC by redirecting workflow traffic away from the scaling region.

We are adding stronger regional health and capacity checks before maintenance and requiring a stable observation period before restoring traffic. We are also improving automated connection resiliency, and partnering with our platform dependency to automatically detect and remediate unhealthy cluster members and shard imbalance. More generally, we already had work underway to improve the resiliency and scale of this piece of Actions infrastructure.

We identified an issue causing delays in GitHub Actions run starts. Some users may have experienced longer than expected wait times when triggering workflow runs. We have applied mitigations and have recovered. Our team continues to monitor and investigate the root cause.

The degradation affecting Actions has been mitigated. We are monitoring to ensure stability.

The issue should now be resolved. Please let us know if you are still experiencing any problems. We appreciate your patience while we worked to resolve this.

Hetzner experienced a network outage at Falkenstein that affected some of our services: https://status.hetzner.com/incident/49567296-8543-43c1-a0e2-9755ad4bd018

All customers have been migrated from affected hardware. If you notice anything out of place, email support@ubicloud.com

We experience problems in a subset of the underlying hardware, we are working with the provider to address these issues.

We are continuing to investigate this issue.

GitHub confirmed that they resolved the issue on their edge servers.

We are continuing to work on a fix for this issue.

We are monitoring the issue with GitHub's servers.

We have traced this to an upstream GitHub issue affecting git data transfer. Connections to GitHub establish normally (DNS, TCP, and TLS are healthy), but transfers stall mid-stream, producing very slow clone and fetch speeds, HTTP/2 stream cancellations, or checkout steps that hang for several minutes. This is occurring across multiple regions and is not specific to Ubicloud infrastructure. The behavior is being reported and tracked by others in the GitHub community here: https://github.com/orgs/community/discussions/196638. We will continue to monitor and update as GitHub addresses the underlying problem. If you need to mitigate in the meantime, the following can help on affected jobs: shallow clones (fetch-depth: 1), forcing HTTP/1.1 (git config --global http.version HTTP/1.1), and adding retry logic around the checkout step.

This incident has been resolved.

Following an initial capacity increase, provisioning times have returned to normal. We're continuing to expand our fleet to meet future demand.