Famous Production Incidents

Real outage case studies with direct answers, root causes, impact, and engineering lessons.

Google Cloud 2019 Outage: Network Control Plane Failure

Google Cloud | Sun Jun 02 2019 00:00:00 GMT+0000 (Coordinated Universal Time)

Google Cloud 2019 Outage: Network Control Plane Failure

A concise incident analysis of the Google Cloud 2019 outage, where a configuration change caused network congestion and broad service impact.

GitHub October 21, 2018 Outage: MySQL Failover

GitHub | 2018-10-21T00:00:00Z

GitHub October 21, 2018 Outage: MySQL Failover

GitHub's October 21, 2018 outage explained: how a network partition and MySQL failover created data-consistency risks, plus recovery decisions and lessons.

How do I find what change caused a production outage?

Start with the first known failure and compare it with recent changes. A nearby deploy is a lead, not proof of the cause. Include configuration, feature flags, infrastructure, and data changes in the investigation.

  1. Establish the timeline. Align the first errors, affected requests, and change events using the same time zone.
  2. Compare affected and healthy systems. Look for differences in versions, configuration, regions, and dependencies.
  3. Test a specific hypothesis. Predict which requests should fail if a candidate change is responsible, then check the evidence.
  4. Verify recovery. Check customer behavior and data correctness after mitigation, not just whether an alert clears.

The GitHub MySQL incident shows why restoring connectivity does not automatically restore correct database state. For a repeatable investigation method, see Google SRE's troubleshooting guide.

What is a production incident?

A production incident is a failure or degradation that affects people using a live service. These case studies explain the impact, cause, and recovery decisions, so you can compare failure patterns across systems.

How can I practice investigating an incident?

Start with our SRE incident response game guide. In The Incident Challenge, you investigate a fictional failure, apply a runtime configuration fix, and validate it with a deploy. The aim is to prove that the system works again.