Incident Response Drills: A 60-Minute Engineering Game Day
By The Incident Challenge Team | 2026-11-04T09:10:00+02:00
Incident response drills let engineering teams practice coordination and technical decisions before customers depend on those decisions. A useful SRE game day has a specific learning objective, a controlled scenario, a facilitator, and a debrief that produces changes someone will own.
This 60-minute plan is an original facilitator pack for a fictional software outage. Run it as a tabletop with prepared evidence or reproduce it in an isolated test environment. Its schedule and thresholds are exercise choices you can change.
Choose one question the drill should answer
Start with a narrow objective: can the team identify a shared dependency, coordinate a bounded mitigation, and prove that delayed work has recovered?
Avoid trying to assess every reliability skill in one session. If the team is new to incident command, simplify the technical puzzle. If coordination is established, introduce a harder correctness or recovery decision.
Google's incident response workbook chapter discusses separating technical resolution from incident coordination and practicing response. Use that distinction when assigning roles in this exercise.
Assign roles and define the boundaries
Use four roles, combining them for a small team:
- Facilitator: holds the scenario, releases evidence, and can stop the drill.
- Incident lead: coordinates priorities, ownership, and decisions.
- Technical responder: investigates and proposes changes.
- Communications owner: records the timeline and gives updates.
Before starting, name the environment, permitted actions, and stop condition. Keep production credentials out of a tabletop. For a runnable exercise, use synthetic accounts and a known reset procedure. Everyone should know that the session is a drill.
Prepare the artifacts in advance: a service map, deployment history, three evidence packets, a mitigation result, and a facilitator answer sheet. Do not improvise contradictory evidence to make the puzzle harder.
The fictional incident: orders are accepted but confirmations lag
The system has an Orders API, a queue, a worker, and a notification dependency. A worker configuration change increases concurrency beyond the dependency's capacity. Timeouts cause repeated delivery attempts, and notification jobs age while API health checks remain green.
Keep that explanation on the facilitator sheet. Tell participants only that customers report missing confirmation emails after successful orders. The order data itself is intact in this scenario.
Run the 60-minute schedule
| Time | Facilitator action | Behavior to observe |
|---|---|---|
| 0–5 minutes | State the objective, roles, environment, and stop condition | Participants understand ownership and scope |
| 5–10 minutes | Release the customer report and initial dashboard | Team defines impact before changing systems |
| 10–20 minutes | Provide deployment history and queue evidence when requested | Responders compare hypotheses and assign checks |
| 20–30 minutes | Release dependency attempt counts and timeout evidence | Team connects repeated work to capacity |
| 30–40 minutes | Ask for a mitigation and expected result | Decision includes tradeoffs and verification |
| 40–45 minutes | Release recovery observations | Team checks delayed work and correctness |
| 45–60 minutes | Run the debrief and assign follow-ups | Findings become concrete improvements |
During the investigation, answer reasonable evidence requests from the prepared packets. If a needed artifact does not exist, say so and record the gap. Do not require participants to guess a particular dashboard name to receive information.
Prepare three coherent evidence packets
Packet A: customer impact. Orders are accepted. Confirmation latency rises. API success rate remains normal. Include an affected time window and a synthetic sample order identifier.
Packet B: recent changes. The worker concurrency setting changed shortly before confirmation latency increased. Queue depth and oldest-message age are rising. No Orders API deployment occurred in the window.
Packet C: failure mechanism. Logical notification jobs arrive at a stable rate, but attempts per job increase. The dependency returns timeouts above the test capacity. Worker logs show repeated attempts for the same synthetic identifiers.
The exercise should permit a responder to disprove an Orders API outage and investigate the worker path. It should also leave room to ask whether duplicates are possible before replaying work.
For deeper individual practice with these signals, use the distributed systems debugging exercise.
Make participants predict recovery before revealing it
Ask for a written prediction: “If the worker setting is the cause, reducing concurrency to the known working limit should lower timeout rate and attempts per job; oldest-message age should then fall as useful completions catch up.”
In a tabletop, show a prepared result consistent with the selected action. In a runnable drill, measure the actual result. If the proposal would only hide failures or discard jobs, explain that consequence and let the team reconsider.
Require a customer-level check, a backlog check, and a correctness check. One successful request does not demonstrate that every delayed job completed. A lower error rate after traffic has been stopped does not demonstrate restored service under demand.
Debrief decisions using the evidence available at the time
Ask each participant which observation most changed their understanding. Then inspect the response process:
- Was an incident lead clear?
- Did two people unknowingly investigate the same thing?
- Were facts, assumptions, and decisions recorded separately?
- Was there an explicit reason to escalate?
- Did verification cover the customer journey and delayed work?
Write one or two changes with an owner, a due date, and a completion check. “Improve monitoring” is too vague. “Add oldest-notification-age to the on-call dashboard and validate it with the next queue drill” gives the team something to complete.
Use the postmortem template to preserve the result. Repeat the relevant skill after the improvements land, using a different scenario rather than the same hidden answer.
Add practice between team drills
A team game day tests how people coordinate around their own service. Individual practice can develop the investigation habits those sessions rely on. The Incident Challenge offers prepared incidents where engineers inspect evidence, adjust runtime configuration, and deploy a fix.
For a progression from individual practice into a rotation, use the on-call training plan.