AWS Kinesis 2020 Outage: When Adding Servers Broke Routing

AWS | 2020-11-25T00:00:00Z

AWS Kinesis 2020 Outage: When Adding Servers Broke Routing

A Kinesis front-end expansion in US-EAST-1 exposed an operating-system thread limit on November 25, 2020. Shard-map construction failed, leaving servers unable to route requests.

Incident answer

Impact: Kinesis requests failed, and dependent services including CloudWatch and Cognito were affected.

Root cause: Peer coordination increased thread use beyond a configured limit, preventing front-end servers from building usable routing maps.

Lesson: Measure coordination costs as a fleet grows, and test cold-start recovery alongside steady-state capacity.

Why did more Kinesis capacity lead to an outage?

AWS's post-event summary distinguishes the capacity addition from the underlying scaling limit. Removing the added servers was insufficient on its own: recovery also required controlled restarts. Bootstrap work competed with customer traffic, so bringing the fleet back too quickly could undermine recovery.

The broader lesson is to examine the resources consumed by coordination, not just the resources available for processing useful work.

A fleet can grow faster than its coordination budget

Consider a fictional design in which every server maintains one dedicated resource for each other server. With 20 servers, each maintains 19 peer resources, for 380 across the fleet. With 40 servers, each maintains 39, for 1,560 across the fleet.

That arithmetic is an illustrative model, not AWS production sizing. It shows how doubling useful capacity can more than double a coordination cost. The exact calculation for your service depends on connection reuse, topology, pooling, and how membership changes propagate.

A capacity review should name the resource and its limit:

Resource Scaling question
Threads or tasks Does each peer, shard, or request add another worker?
Connections Does each process maintain a separate pool to every destination?
Routing metadata How much state must a joining instance obtain?
Membership updates Does a fleet change generate work on all existing members?
Bootstrap capacity Can new instances become ready while serving demand?

CPU and memory remain useful signals, but they cannot substitute for a resource the design constrains independently.

Treat joining the fleet as a workload

For your own system, distinguish starting a process from making it useful. A process may need configuration, credentials, a routing map, a cache, or a replayed log before it can handle representative requests.

Measure that preparation under the conditions in which recovery will happen. An isolated startup benchmark may miss contention with customer traffic. A warm restart may miss work required after losing local state.

Write an admission condition that reflects readiness. Then check whether the health system could remove a recovering instance before it completes initialization. A responder needs to understand that interaction before attempting a fleet-wide restart.

A practice exercise: warm capacity versus cold capacity

Use a disposable service or a tabletop with synthetic observations. Provide a front end that needs a routing map before it can send work to a small worker pool. Give the map-loading path a bounded capacity.

Run three versions of the exercise:

  1. Start one instance with no customer demand.
  2. Start one instance while synthetic demand is present.
  3. Start several instances together under the same demand.

Record preparation time, useful completions, failed requests, and resources consumed by the initialization path. If running a tabletop, prepare internally consistent results for those conditions rather than inventing new numbers during the session.

Ask the responder to choose a recovery batch size and describe what would make them pause. The objective is to identify a sustainable recovery rate, not merely the fastest way to launch processes.

Verify the service that depends on the service

A useful recovery review includes downstream operations and accumulated work. Check whether delayed events remain accounted for and whether monitoring has resumed. A quiet graph may reflect missing observations rather than healthy behavior.

For your own dependencies, identify which failures can be buffered, which can degrade gracefully, and which must fail the customer request. Exercise the boundary where a supposedly optional dependency stops behaving as optional.

Document the decision with the postmortem template. Compare the failure pattern with AWS's 2025 DynamoDB DNS incident, which illustrates a different route to a broad dependency failure.

Source and further practice