Ubicloud - Postgres DNS resolution issues – Incident details

Postgres DNS resolution issues

ResolvedPartial outage2 hours 13 minutes

Affected

PostgreSQL

Partial outage from 8:00 PM to 10:13 PM

  • ResolvedUTC

    Cross checking DNS records has been successful for some time. Customers report negative caches have since expired. Likely cause: race condition that caused misapplication of DNS records that was exposed under higher parallelism. We have to analyze it and patch it in the coming days.

  • MonitoringUTC

    We've identified this problem, and believe negative caching of DNS records to be all that remains. We are going to continue to check for at least thirty minutes, the longest time reported for negative caching so far.

  • IdentifiedUTC

    Some records are proving more difficult to address than we expected.

  • MonitoringUTC

    We've hotfixed the remaining records. Negative caching of DNS can slow recovery, depending on the upstream DNS server. We will remain in contact with all affected customer until all symptoms are resolved.

  • InvestigatingUTC

    There is a defect in Postgres that causes unbound DNS records. Negative caching can exacerbate the incident after our fix. Databases affected: many at the outset, some now. Diagnosing the last records.