SRE Training Exercises: 5 Scenarios With Success Criteria
By The Incident Challenge Team | 2026-11-04T09:00:00+02:00
SRE training exercises should give engineers a user-visible problem, enough evidence to investigate it, and a way to prove that their fix worked. A useful exercise ends with a recovered service and an explanation of the failure mechanism.
The five scenarios below are original exercise designs. Run them in an isolated test environment, or present the evidence as a tabletop. The time boxes are suggested facilitation choices, not industry benchmarks.
Set up the exercise before revealing the cause
Give the participant a service map, an initial alert, a recent-change history, and access to the relevant logs or recorded evidence. Keep the intended explanation in a separate facilitator sheet. If the briefing names the broken component, much of the investigation has already been done for them.
Record four things: the first hypothesis, the observation that changed it, the mitigation selected, and the recovery check. These make a stronger debrief than a completion time alone.
Google's emergency response chapter describes the importance of preparing and practicing for incidents. The exercises here put that principle into concrete situations your team can adapt.
1. A retry storm behind a healthy-looking average
Suggested time: 20 minutes. Skill: separating original demand from repeated work.
In your test setup, route a checkout request through an API and an inventory dependency. Make a small fraction of inventory calls exceed the caller's timeout. Configure a bounded retry policy for the exercise, then compare it with retries disabled. Do not run an uncontrolled load generator against a shared service.
Hand the responder these starting observations:
- Customer request rate is approximately flat.
- Inventory attempt rate has increased.
- Average dependency latency is close to normal, while tail latency has increased.
- Checkout failures started after a timeout or retry configuration change.
Ask them to identify where extra work enters the system. A useful investigation compares logical requests with attempts and traces the deadline across service boundaries.
Success criteria: the responder explains the amplification, chooses a bounded mitigation, and shows that customer success rate improves without an unacceptable increase in dependency work. Simply increasing every timeout does not establish recovery.
2. Stale reads after a successful write
Suggested time: 20 minutes. Skill: distinguishing availability from correctness.
Create a toy profile service with a primary data store and a deliberately delayed read model. An update returns success, but a subsequent profile read shows the old value. Provide timestamps, record versions, and the read-routing configuration.
The responder should compare the acknowledged write with the version returned by each read path. Encourage them to state the consistency requirement before proposing a change: does this particular workflow require read-after-write behavior?
Success criteria: the explanation identifies the stale path, the mitigation respects the workflow's consistency needs, and verification checks both the changed record and an unaffected record. Restarting an API process is insufficient evidence of a data fix.
For a real case involving difficult database recovery decisions, read GitHub's 2018 MySQL incident.
3. Queue depth falls while customers still wait
Suggested time: 25 minutes. Skill: measuring useful completion.
Build a disposable queue with synthetic tasks. Have a worker fail and requeue one class of messages. Include a dead-letter path or explicit failure count so messages cannot silently disappear.
Give the participant queue depth, oldest-message age, attempt counts, completed-task counts, and worker errors. Add a second worker and show that aggregate depth briefly improves while the oldest affected task remains stuck.
Success criteria: the responder distinguishes throughput from retries, identifies the failing message class, and demonstrates that valid tasks complete and failed tasks remain accounted for. An empty queue achieved by discarding work is not recovery.
4. One instance has a different configuration
Suggested time: 20 minutes. Skill: finding failures hidden by aggregation.
Run several local replicas of a small service. Give one replica a different dependency endpoint or feature setting. Route synthetic requests across them and attach an instance identifier and configuration revision to each response or log event.
The initial dashboard should show intermittent failures across one endpoint. Let the responder choose how to group the evidence. Instance, version, and configuration revision should reveal more than a fleet-wide error average.
Success criteria: the responder finds the inconsistent replica, identifies the exact setting, and verifies the intended configuration across every replica after remediation. Read Knight Capital's deployment failure for a historical example of why fleet consistency matters.
5. The backup exists, but restoration is unproven
Suggested time: 30 minutes. Skill: validating recovery assumptions.
Use a disposable database containing synthetic records. Supply a backup, a restore procedure, and a known set of expected records. Introduce one controlled obstacle, such as an outdated tool instruction or a missing restore permission.
Ask the responder to restore into a separate destination, identify the recovery point, and validate the application against it.
Success criteria: the team can identify what was recovered, what remains missing, and how long the complete restore took. A successful backup job alone does not answer those questions. GitLab's 2017 database incident is a useful companion case.
Score evidence and recovery separately
Use a simple rubric for each scenario:
| Dimension | Evidence to look for |
|---|---|
| Impact | Names the affected customer operation and scope |
| Investigation | Tests a hypothesis and records contrary evidence |
| Mitigation | Explains the expected benefit and tradeoff |
| Recovery | Checks customer behavior, correctness, and remaining work |
| Learning | Proposes a specific improvement with an owner |
If someone finds the cause quickly but cannot prove recovery, that is a useful training result. If someone needs a hint but then makes a careful, justified fix, preserve that distinction too.
Turn individual practice into team readiness
Use the on-call training checklist to place these exercises in an onboarding plan. For a group session, use the 60-minute incident response drill.
For a prepared investigation environment, try a past Incident Challenge. Players work through in-app evidence, change runtime configuration, and deploy a fix. The five exercises above are suggestions for your own training setup, rather than a list of challenges promised in the product.