On-Call Training for Engineers: A Practical Readiness Checklist

On-Call Training for Engineers: A Practical Readiness Checklist

By The Incident Challenge Team | 2026-11-04T09:05:00+02:00

On-call training for engineers should establish that someone can recognize customer impact, investigate safely, call for help, and verify a mitigation. Reading a runbook is preparation; demonstrating those actions is evidence of readiness.

The checklist and four-week plan below are a proposed starting point for a software engineering team. Adapt the pace to the service's risk and the engineer's experience. A date on a rota should not substitute for demonstrated readiness.

Define the job before teaching the tools

Write down what the primary responder owns. For example: acknowledge a page, assess impact, open an incident when required, start a timeline, and involve the appropriate service owner. Specify who can authorize risky changes and who takes over if the responder is stuck.

If the service handles money, customer data, or irreversible work, make those boundaries concrete. A new responder needs to know which actions can be reversed and which need another person involved under your team's procedures.

Google's SRE onboarding chapter describes approaches to building operational knowledge, including practical learning and incident exercises. The plan here uses that general approach, with explicit checks for your own environment.

Before the first shadow shift: prove access

Have the engineer demonstrate the following during normal working hours:

Use a test page and read-only inspection where possible. For permissions that can only be exercised through a change, demonstrate them in an approved test environment. Document missing access as a training blocker with an owner, rather than leaving it for the first night shift.

Week 1: follow a request through the service

Choose one critical user journey, such as signing in or submitting an order. Ask the engineer to draw its path through the entry point, application services, data stores, and asynchronous work.

Then ask three questions at each boundary:

  1. What will the customer experience if this dependency slows down?
  2. Which signal distinguishes a slow dependency from a slow caller?
  3. What is the safest available mitigation, and what does it cost?

The deliverable is a short service map connected to real dashboards and runbooks. It does not need to describe every component in the company. It needs to support the user journey the engineer will help protect.

Week 2: shadow decisions, then explain them

Pair with an experienced responder for real pages when available. For quiet periods, replay a historical incident with the ending hidden.

After each decision, ask the trainee to explain the evidence available at that moment. Avoid judging an early decision using facts discovered much later. Record uncertainty explicitly: a deployment may be suspicious without being proven causal.

A useful shadow note contains:

Observation Working explanation Next check Reason to escalate
Checkout failures rose in one region A regional dependency may be degraded Compare dependency errors across regions Impact is growing or no safe mitigation is available

By the end of the week, the engineer should be able to write a brief handover another responder can continue without repeating the investigation.

Week 3: reverse shadow with a bounded exercise

Let the trainee lead while the experienced responder observes. Start with a synthetic incident or an isolated environment. Select a scenario from these SRE training exercises, such as configuration drift or a stuck queue.

Include one moment where escalation is the right action. For example, the apparent mitigation may risk duplicate writes, or the responder may lack evidence to decide whether a replica is safe to promote. Asking for help should count as good judgment when the situation calls for it.

The observer records actions and reasoning, then gives feedback after the exercise. If intervention is necessary to keep the exercise within its agreed boundaries, record the intervention and repeat the relevant step later.

Week 4: demonstrate readiness on a fresh scenario

Use a different failure mechanism so the engineer cannot pass by remembering the previous answer. Change the symptom, affected path, or misleading signal.

Review readiness against observable behavior:

Capability Ready when the engineer can…
Triage State the affected user operation and current uncertainty
Investigation Connect a hypothesis to a specific observation
Mitigation Explain the expected effect and recovery risk
Escalation Reach the correct person with a useful summary
Communication Give a factual update with a next checkpoint
Verification Check customer outcomes and remaining backlog or data risk
Handover Transfer ownership, evidence, and pending actions clearly

There is no universal passing time for these capabilities. Agree on service-specific expectations before the session. If one area needs work, keep the engineer paired for that responsibility and give them another opportunity to demonstrate it.

Give responders a usable update format

Practice a message like this with fictional data:

Checkout requests in the test region are failing after the inventory change. We are comparing configuration revisions and have not confirmed the cause. The service owner is joining. Next update in ten minutes, or sooner if impact changes.

The message distinguishes what is known from what is being investigated. It also tells the next responder where to help.

Keep training current after onboarding

Revisit the checklist after major architecture changes and whenever a runbook fails during an incident. Use a team incident drill to exercise coordination, then record improvements with the incident postmortem template.

For individual investigation practice between shifts, try a past Incident Challenge. It provides an additional way to practice reading evidence, correcting runtime configuration, and deploying a fix. Your own service access, escalation paths, and recovery procedures still need to be practiced in your team's environment.