Ubicloud - Notice history

Compute - Operational

100.0% uptime
Sep 2026 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2026
Oct 2026
Nov 2026

PostgreSQL - Operational

100.0% uptime
Sep 2026 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2026
Oct 2026
Nov 2026

Networking - Operational

100.0% uptime
Sep 2026 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2026
Oct 2026
Nov 2026

Kubernetes - Operational

100.0% uptime
Sep 2026 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2026
Oct 2026
Nov 2026

Notice history

View current status

Nov 2026

Incident with Pull Requests
ResolvedDegraded performance4 hours 26 minutes
  • ResolvedUTC

    This incident has been resolved. Thank you for your patience and understanding as we addressed this issue. A detailed root cause analysis will be shared as soon as it is available.

  • UpdateUTC

    We’ve mitigated the issue affecting pull request merges and Actions workflow start times by reducing load on affected systems. Pull Requests and Actions have recovered, and we’re continuing to monitor their performance.

  • MonitoringUTC

    The degradation affecting Actions and Pull Requests has been mitigated. We are monitoring to ensure stability.

  • UpdateUTC

    We’re seeing early signs of improvement in pull request merge times & workflow start times, but recovery is not yet confirmed. Users may still have difficulty merging pull requests. We’re reducing load on affected systems and monitoring the impact. We’ll share another update as we learn more.

  • UpdateUTC

    We’re seeing early signs of improvement in pull request merge times, but recovery is not yet confirmed. The last confirmed timeout rate was 2.5%, compared with a baseline near 0.2%, so some users may still have difficulty merging pull requests. We’ll share another update as we learn more.

  • InvestigatingUTC

    Actions is experiencing degraded performance. We are continuing to investigate.

Slow or stalled git checkouts on GitHub Actions runners
ResolvedDegraded performance8 hours 27 minutes
  • ResolvedUTC

    Git checkouts on GitHub Actions runners are back to normal. The fix is deployed to all runner hosts.

    Cause: Runners in our European data centers were sometimes sent to GitHub servers in the US instead of the nearby ones in Europe. Some of those long-distance connections were losing packets, so git fetches became very slow. Runners now look up github.com through a DNS provider that picks GitHub servers based on where the request comes from, so each runner connects to the GitHub servers closest to it.

  • IdentifiedUTC
    Some GitHub Actions jobs on Ubicloud runners hang during actions/checkout or other git fetches from github.com. The download starts but then runs very slowly, often for 15 to 60 minutes, until the job times out. We have identified the cause and are rolling out a fix.

Oct 2026

Incident with Actions, Pages, and Webhooks
ResolvedPartial outage1 hour 32 minutes
  • ResolvedUTC

    On October 30, 2026, from 17:50 to 18:32 UTC, Actions, Pages, and Webhooks experienced degraded performance. Actions workflow processing, Pages deployments, and webhook deliveries were delayed, and code search and live updates returned errors. Some Actions queues took additional time to drain after mitigation, and the incident was resolved at 19:23 UTC.

    The impact was triggered by maintenance on a cloud network gateway connecting an Actions service in the GitHub data center to Redis clusters in the Central US Azure region. The service's Redis client retried aggressively and redialed those connections. That produced a connection storm that saturated the gateway, with about 90% of connection attempts failing. Other services sharing the same network path were degraded as well.

    We mitigated the incident by removing the affected resources from rotation, which ended the connection storm and allowed the gateway to recover. We have already reduced the number of connection retries in the impacted service and are further improving retry behavior with jittered backoff, shorter connect timeouts, and retry budgets. We are also working with our infrastructure team to make GitHub more resilient to gateway maintenance.

  • MonitoringUTC

    The degradation affecting Actions, Pages and Webhooks has been mitigated. We are monitoring to ensure stability.

  • UpdateUTC

    All affected services have recovered following mitigation. We continue to investigate root cause.

  • UpdateUTC

    We continue to investigate delays affecting Actions workflows, Pages deployments, and webhook events, along with errors affecting code search and live updates. We have taken mitigation steps. Actions processing and Pages deployments are showing signs of recovery, but impact has not fully resolved. We are monitoring recovery and investigating the underlying cause.

  • UpdateUTC

    Pages is experiencing degraded performance. We are continuing to investigate.

  • InvestigatingUTC

    We are investigating reports of degraded performance for Actions and Webhooks

Increased provisioning times for GitHub runners
ResolvedDegraded performance1 hour 56 minutes
  • UpdateUTC

    We investigated the cache slowness. On October 29, Cloudflare had issues on several of their services. From approximately 09:00 to 22:00 UTC, Cloudflare sent our cache traffic to its Prague location instead of Frankfurt, and Cloudflare R2 had increased latency. Because of this, cache upload and download speeds were much lower than normal.

    Cache performance has been normal since approximately 23:00 UTC on October 29.
    We are working with Cloudflare to find the root cause, and to prevent and detect similar problems earlier in the future.

  • ResolvedUTC

    Wait times for GitHub Actions runners are back to normal.

    Cause: Today we saw a surge in demand that put pressure on our capacity at peak times, especially while EU evenings and US mornings overlapped. This week the overlap is an hour longer than usual because Europe has already switched to winter time and the US hasn't yet. Slowness in Cloudflare R2, which we use for cache storage, also made jobs run longer. This further added to queue times.

    We're adding capacity to handle a continuous increase in overall demand. We apologize for the delay in processing times.

  • MonitoringUTC

    Wait times are improving. We're processing the remaining backlog and monitoring until queue times are back to normal.

  • InvestigatingUTC

    We're seeing longer than usual wait times for GitHub Actions runners to start, due to high demand. We're investigating and will post updates here.

Incident with Actions, Issues, API Requests and Pull Requests
ResolvedDegraded performance8 minutes
  • ResolvedUTC

    On October 26, 2026, between 17:52 and 18:49 UTC, some customers experienced slower response times across the Checks API, Issues, Pull Requests, GraphQL, Notifications, and related services. Impact was limited to requests served from one of our datacenter sites. Some GraphQL requests failed, and some notifications were delayed. Among GraphQL requests served from the affected site, the average error rate was 5.68%, with a peak five-minute error rate of 14.32%.

    Unusually large requests for contribution history overwhelmed part of our database infrastructure, causing a backlog. Requests continued reaching the slow component rather than being redirected to healthy systems. Restarting the component restored its performance. The incident was marked mitigated at 18:46 UTC and resolved at 18:57 UTC.

    We are adding limits to these requests, improving alerts to detect overload sooner, and strengthening automatic redirection to healthy systems.

  • UpdateUTC

    We saw elevated latency affecting REST API requests, including Checks, and some Actions, Issues, Pull Requests, and GraphQL functionality. Some requests may have been slow or timed out, and notifications were delayed from 17:59 UTC - 18:40 UTC.

    We mitigated the issue and all services have recovered.

  • MonitoringUTC

    The degradation has been mitigated. We are monitoring to ensure stability.

  • InvestigatingUTC

    We are investigating reports of degraded performance for API Requests, Actions, Issues and Pull Requests

Sep 2026

Increased provisioning times for large runner sizes (16 and 30 vCPUs)
ResolvedDegraded performance1 hour 45 minutes
  • PostmortemUTC

    Between approximately 15:00 UTC and 17:00 UTC on September 23, jobs took longer than usual to start, and some jobs were cancelled mid-run. Overall demand for our runners exceeded our dedicated capacity, and larger runners (16 and 30 vCPUs) were hit hardest, since they're the hardest to place when capacity is tight.

    To absorb the extra demand, we ran some jobs on external compute resources. Some of those machines were reclaimed mid-run, which cancelled their jobs. Handling the lost machines slowed down our control plane, so jobs on our own servers also took longer to start. The more demand grew, the more interruptions we saw, and the further provisioning fell behind.

    We stopped using reclaimable external compute resources and added more dedicated capacity. We're also changing how our control plane handles unreachable machines, so problems with external compute resources can't slow down provisioning on our own servers. This incident is resolved.

  • ResolvedUTC

    All systems are operating normally.

  • InvestigatingUTC

    We're seeing higher than usual demand for our runners. This is causing longer than usual provisioning times for larger runners (16 and 30 vCPUs). We're working to increase our capacity as quickly as possible and mitigate the issue.

Incident across several services
ResolvedDegraded performance18 hours 44 minutes
  • ResolvedUTC

    Starting at 07:57 UTC on September 23, GitHub experienced elevated 500 and 404 responses across several application pages. This caused failures when installing GitHub Apps, creating organizations, and making some organization membership changes. Customers also experienced delayed label updates and stale search results in Projects.

    The infrastructure failure was isolated to our Azure Central US region. The API errors were mitigated by 10:58 UTC on September 23. Projects' processing continued to recover while an accumulated backlog was drained, and full service was restored at 04:55 UTC on September 24.

    The incident was caused by a failed planned maintenance operation on a primary database. Automated recovery initiated an emergency database failover, after which several replicas in the affected region were unable to resume replication correctly. This reduced available database capacity and caused the API errors and downstream Projects processing delays.

    We have mitigated the immediate failure mode. We are also improving maintenance safety checks, database failover handling, post-failover replica validation, and downstream processing resilience to reduce the likelihood and impact of similar incidents.

  • MonitoringUTC

    The degradation has been mitigated. We are monitoring to ensure stability.

  • UpdateUTC

    We are continuing to process the backlog of issue label updates for Projects. Label changes may still be delayed. All other services are operating normally.

  • UpdateUTC

    We are continuing to process the backlog of issue label updates for Projects. Users may still see delays before label changes are reflected in Projects. All other services are operating normally.

  • UpdateUTC

    We've deployed a change intended to accelerate processing of the backlog of issue label updates in Projects. A sizable backlog still remains and we continue working through it. All other services are operating normally. We will provide another update within the next hour.

  • UpdateUTC

    Updates to issue labels may be delayed in being reflected in Projects. We are continuing to deploy a change that will accelerate processing of the backlog of label updates. All other services are available. We will provide another update within the next hour.

  • UpdateUTC

    We are preparing to deploy a change that will mitigate the impact.

  • UpdateUTC

    Continuing to investigate the lag that may be experienced in issue labels being accurately reflected in Projects. We are working on alternate solutions to process the backlog of label updates.

  • UpdateUTC

    We will post another update in approximately one hour to share our progress.

  • UpdateUTC

    Updates to issue labels may be delayed in being reflected in Projects by about ~10 minutes. We have added some capacity to work through the backlog more quickly, but it'll likely be a few hours to complete processing the full backlog of messages. All other services are available.

  • UpdateUTC

    Users may experience stale Project search results. We are working to increase indexing speed. All other services are available.

  • UpdateUTC

    We are seeing recovery for Projects. Users may experience stale search results for Projects while indexing catches up.

  • UpdateUTC

    The degradation affecting API Requests has been mitigated. We are monitoring to ensure stability.

  • UpdateUTC

    Database replicas have been restored. Org creation and the GitHub API are no longer degraded.

  • UpdateUTC

    Database replicas have detached. We're working to restore the database replicas. Users may experience issues beyond creating organizations and a degraded experience with the GitHub API and Projects.

  • InvestigatingUTC

    We are investigating reports of degraded performance for API Requests

Previous

Sep 2026 to Nov 2026

Next