Ubicloud - Postgres DNS resolution issues – Incident details

All systems operational

Postgres DNS resolution issues

Resolved
Partial outage
Started 8 months agoLasted 2 hours 13 minutes

Affected

PostgreSQL

Partial outage from 8:00 PM to 10:13 PM

Updates
  • Resolved
    UTC
    Resolved

    Cross checking DNS records has been successful for some time. Customers report negative caches have since expired. Likely cause: race condition that caused misapplication of DNS records that was exposed under higher parallelism. We have to analyze it and patch it in the coming days.

  • Monitoring
    UTC
    Monitoring

    We've identified this problem, and believe negative caching of DNS records to be all that remains. We are going to continue to check for at least thirty minutes, the longest time reported for negative caching so far.

  • Identified
    UTC
    Identified

    Some records are proving more difficult to address than we expected.

  • Monitoring
    UTC
    Monitoring

    We've hotfixed the remaining records. Negative caching of DNS can slow recovery, depending on the upstream DNS server. We will remain in contact with all affected customer until all symptoms are resolved.

  • Investigating
    UTC
    Investigating

    There is a defect in Postgres that causes unbound DNS records. Negative caching can exacerbate the incident after our fix. Databases affected: many at the outset, some now. Diagnosing the last records.