On-Call Training for Engineers: A Practical Readiness Checklist
By The Incident Challenge Team | 2026-11-04T09:05:00+02:00
On-call training for engineers should establish that someone can recognize customer impact, investigate safely, call for help, and verify a mitigation. Reading a runbook is preparation; demonstrating those actions is evidence of readiness.
The checklist and four-week plan below are a proposed starting point for a software engineering team. Adapt the pace to the service's risk and the engineer's experience. A date on a rota should not substitute for demonstrated readiness.
Define the job before teaching the tools
Write down what the primary responder owns. For example: acknowledge a page, assess impact, open an incident when required, start a timeline, and involve the appropriate service owner. Specify who can authorize risky changes and who takes over if the responder is stuck.
If the service handles money, customer data, or irreversible work, make those boundaries concrete. A new responder needs to know which actions can be reversed and which need another person involved under your team's procedures.
Google's SRE onboarding chapter describes approaches to building operational knowledge, including practical learning and incident exercises. The plan here uses that general approach, with explicit checks for your own environment.
Before the first shadow shift: prove access
Have the engineer demonstrate the following during normal working hours:
- Receive and acknowledge a test page.
- Open the incident communication channel and shared timeline.
- Find customer-facing availability and latency measurements.
- Read relevant logs and recent deployment history.
- Find the current service owner and backup escalation contact.
- Locate the rollback or configuration-change procedure.
- Identify the permissions required for recovery actions.
Use a test page and read-only inspection where possible. For permissions that can only be exercised through a change, demonstrate them in an approved test environment. Document missing access as a training blocker with an owner, rather than leaving it for the first night shift.
Week 1: follow a request through the service
Choose one critical user journey, such as signing in or submitting an order. Ask the engineer to draw its path through the entry point, application services, data stores, and asynchronous work.
Then ask three questions at each boundary:
- What will the customer experience if this dependency slows down?
- Which signal distinguishes a slow dependency from a slow caller?
- What is the safest available mitigation, and what does it cost?
The deliverable is a short service map connected to real dashboards and runbooks. It does not need to describe every component in the company. It needs to support the user journey the engineer will help protect.
Week 2: shadow decisions, then explain them
Pair with an experienced responder for real pages when available. For quiet periods, replay a historical incident with the ending hidden.
After each decision, ask the trainee to explain the evidence available at that moment. Avoid judging an early decision using facts discovered much later. Record uncertainty explicitly: a deployment may be suspicious without being proven causal.
A useful shadow note contains:
| Observation | Working explanation | Next check | Reason to escalate |
|---|---|---|---|
| Checkout failures rose in one region | A regional dependency may be degraded | Compare dependency errors across regions | Impact is growing or no safe mitigation is available |
By the end of the week, the engineer should be able to write a brief handover another responder can continue without repeating the investigation.
Week 3: reverse shadow with a bounded exercise
Let the trainee lead while the experienced responder observes. Start with a synthetic incident or an isolated environment. Select a scenario from these SRE training exercises, such as configuration drift or a stuck queue.
Include one moment where escalation is the right action. For example, the apparent mitigation may risk duplicate writes, or the responder may lack evidence to decide whether a replica is safe to promote. Asking for help should count as good judgment when the situation calls for it.
The observer records actions and reasoning, then gives feedback after the exercise. If intervention is necessary to keep the exercise within its agreed boundaries, record the intervention and repeat the relevant step later.
Week 4: demonstrate readiness on a fresh scenario
Use a different failure mechanism so the engineer cannot pass by remembering the previous answer. Change the symptom, affected path, or misleading signal.
Review readiness against observable behavior:
| Capability | Ready when the engineer can… |
|---|---|
| Triage | State the affected user operation and current uncertainty |
| Investigation | Connect a hypothesis to a specific observation |
| Mitigation | Explain the expected effect and recovery risk |
| Escalation | Reach the correct person with a useful summary |
| Communication | Give a factual update with a next checkpoint |
| Verification | Check customer outcomes and remaining backlog or data risk |
| Handover | Transfer ownership, evidence, and pending actions clearly |
There is no universal passing time for these capabilities. Agree on service-specific expectations before the session. If one area needs work, keep the engineer paired for that responsibility and give them another opportunity to demonstrate it.
Give responders a usable update format
Practice a message like this with fictional data:
Checkout requests in the test region are failing after the inventory change. We are comparing configuration revisions and have not confirmed the cause. The service owner is joining. Next update in ten minutes, or sooner if impact changes.
The message distinguishes what is known from what is being investigated. It also tells the next responder where to help.
Keep training current after onboarding
Revisit the checklist after major architecture changes and whenever a runbook fails during an incident. Use a team incident drill to exercise coordination, then record improvements with the incident postmortem template.
For individual investigation practice between shifts, try a past Incident Challenge. It provides an additional way to practice reading evidence, correcting runtime configuration, and deploying a fix. Your own service access, escalation paths, and recovery procedures still need to be practiced in your team's environment.