Reddit | 2023-03-14T00:00:00Z
Reddit's 2023 Kubernetes Outage: A Node Label Broke Routing
Reddit's March 14, 2023 outage followed a Kubernetes upgrade that removed a node label used by its Calico route reflector configuration. Routing failed, and recovery required restoring the cluster.
Incident answer
Impact: The 314-minute outage affected many core Reddit user flows.
Root cause: Calico route reflector selectors depended on a node label removed during the Kubernetes 1.24 upgrade.
Lesson: Test upgrades against infrastructure selectors and hidden dependencies, and rehearse the recovery procedure.
What happened on Reddit's Pi Day outage?
Reddit's engineering postmortem describes a 314-minute outage on March 14, 2023. An upgrade from Kubernetes 1.23 to 1.24 removed the node-role.kubernetes.io/master label used by Calico route reflector selectors. The routing dependency was not apparent during the initial investigation.
The team restored the cluster from a backup before identifying the subtle cause. The report also describes an outdated restore procedure and infrastructure differences between clusters. Those facts make this a useful case for reviewing upgrade assumptions and recovery readiness. Source: Reddit Engineering.
A label can be a production contract
The engineering implication is broader than any one Kubernetes version: metadata becomes operational behavior when another component selects on it. A label can determine where routing runs, where a workload is placed, or whether a policy applies.
Treat such dependencies as contracts that need an owner and a test. For a planned metadata change, ask which selectors consume the value, whether an empty match is valid, and what user-visible behavior would fail if the match disappeared.
This is different from checking that a configuration file is syntactically valid. A valid selector can select nothing. A component can start successfully while being connected to the wrong set of peers.
Trace the dependencies beneath an apparently healthy workload
For an upgrade exercise in your own environment, draw a critical path with at least three layers:
| Layer | Question to investigate |
|---|---|
| Customer workflow | Can a representative request complete correctly? |
| Application dependencies | Can the services reach the stores and peers they require? |
| Platform behavior | Do selectors, routes, admission rules, and discovery still produce the intended result? |
Record an expected observation at each layer. Application health checks are useful, but they may not exercise every network path or every dependency required for a customer operation.
When results differ between environments, inspect the differences rather than assuming an earlier successful rollout covers the later one. The relevant difference may be topology or a rarely used infrastructure component, even when the application version matches.
A practice exercise: the selector still parses
Create a disposable training environment, or use a set of static configuration and observation files for a tabletop. Define a toy routing component selected by a node label. Provide before-and-after metadata in which the label changes while the selector remains unchanged.
Start participants with an application connectivity failure, a recent platform upgrade, and several healthy-looking process checks. Keep the selector mismatch in the facilitator notes.
Ask the responder to establish:
- Which user path is affected and which path still works.
- Which selected node set existed before and after the change.
- Whether the routing or discovery result matches the intended topology.
- What evidence would show that a proposed compatibility change restores behavior.
- How to return the exercise environment to a known state.
Success means explaining the dependency and verifying the resulting connectivity. Guessing that “Kubernetes is broken” does not identify the contract that changed.
This is an original exercise inspired by the failure pattern, rather than a reproduction of Reddit's production infrastructure.
Make the recovery path independently reviewable
For your own upgrade plan, review the recovery procedure with someone who did not write it. Confirm which versions and tools it assumes, which resources it replaces, and which state must remain consistent.
Then execute the procedure in a suitable test environment and record what worked. If an instruction has to be rewritten, update the runbook and repeat the affected step. A document becomes stronger evidence when another person can use it successfully.
Use the on-call readiness checklist to include this skill in onboarding. Use the incident response drill plan to test the handover and coordination around it.
For individual evidence-based investigations, try a past Incident Challenge.