GitLab 2017 Database Outage: Deletion, Backups, and Recovery

GitLab | 2017-01-31T00:00:00Z

GitLab 2017 Database Outage: Deletion, Backups, and Recovery

GitLab.com's January 31, 2017 outage involved accidental deletion on the primary PostgreSQL server during replication troubleshooting. Recovery relied on an earlier snapshot, leaving some database changes unrecoverable.

Incident answer

Impact: GitLab.com was unavailable for many hours and lost some database changes; repository and wiki contents were not lost.

Root cause: A destructive operation intended for a secondary ran on the primary, while backup and recovery failures limited restoration options.

Lesson: Validate database recovery end to end and make the target of destructive operations unmistakable.

What happened at GitLab in 2017?

On January 31, 2017, GitLab.com suffered a database outage after an engineer accidentally removed data from the primary PostgreSQL server while troubleshooting replication. GitLab's final postmortem separates the accidental deletion from the recovery failures that made the incident more damaging.

The team restored from an LVM snapshot taken earlier that day. Some intervening changes to database-backed content were unrecoverable. Git repositories and wiki contents were not lost, although access was affected during the outage. GitLab's initial incident account documented the unfolding recovery; the later postmortem provides the fuller account.

The sequence behind the headline

The final report describes database load, replication lag, and a secondary that needed to be resynchronized. During that work, a command intended to clear the secondary's data directory ran against the primary instead.

Recovery then exposed a separate problem: expected logical backups were missing because the backup tool's version was incompatible with the database. The earlier snapshot became the practical recovery path. That sequence matters because preventing the initial operation and improving recovery are distinct engineering tasks. Source: GitLab's postmortem.

Why a backup job is only one part of recovery

The broader engineering question is whether a team can recover an application from the assets it actually has. Consider the chain of dependencies:

  1. A backup must finish and produce usable data.
  2. Someone must be able to locate and decrypt it.
  3. The restore environment must have compatible tools and permissions.
  4. The restored database must represent a known recovery point.
  5. The application must work against that restored state.
  6. Missing or repeated work must be identified and handled.

A green scheduler entry only addresses a small part of that chain. A file's existence is stronger evidence, but still does not demonstrate a working restore.

For your own service, write down two separate objectives: how long recovery may take, and how much recent data the business can tolerate losing. Then measure an actual restore against them. Avoid promising a recovery point based only on how often a scheduled job is supposed to run.

Questions for a database incident review

Use this case to inspect the conditions around high-impact maintenance:

These are suggested review questions, rather than claims about GitLab's current architecture. A useful answer should point to a demonstrable control or a known gap.

A practice exercise: restore and explain the missing window

Create a disposable database with synthetic records. Take a backup, add a second batch of records, and make the original database unavailable within the exercise environment. Give a responder the backup location, a restore runbook, and expected record counts.

Ask them to restore into a separate destination and answer:

Question Evidence required
What recovery point did we reach? Backup timestamp plus recovered records
Does the application work? A representative read and write against the restored copy
What is missing? Comparison with the synthetic source-of-truth record list
Can another responder repeat this? A corrected runbook and a second walkthrough

Do not score an empty but reachable database as success. The exercise is complete when the responder can explain both the recovered state and its limits.

Record the result with the incident postmortem template. For more practice formats, use the SRE training exercises or try a past Incident Challenge.

Sources and related incidents