Incident Response Runbook Template With a Worked Example

Incident Response Runbook Template With a Worked Example

By The Incident Challenge Team | 2026-11-08T16:21:16Z

An incident response runbook should tell a responder when a procedure applies, how to confirm the target, what action to take, when to stop, and how to prove recovery. Give each step an expected observation and a clear path when that observation is absent.

The template and example below are original working materials for software teams. Adapt them to a specific service and validate the procedure in an appropriate test environment.

AWS's runbook guidance emphasizes a defined outcome, required tools and permissions, error handling, ownership, and validation by another team member. Those are useful foundations for a procedure someone must follow under pressure.

Copy this incident response runbook template

Runbook title and identifier:
Service / environment / owner:
Last verified date and tested software or configuration revision:

Use this procedure when
- Confirmed symptom:
- Scope and prerequisites:
- Situations that require a different procedure:

Before changing anything
- Confirm the environment and exact target identifier.
- Record the current version and relevant evidence.
- Verify required access and the recovery path.
- Name the action owner and the person observing the result.

Steps
1. Action:
   Expected observation:
   If the observation differs:
2. Action:
   Expected observation:
   If the observation differs:

Stop conditions
- Unexpected target or revision:
- Missing prerequisite or evidence:
- Worsening customer impact or data risk:

Recovery checks
- Representative customer operation:
- Remaining backlog or data-correctness checks:
- Observation window and owner:

Escalation / handover
- Contact and backup:
- Evidence, actions, and unresolved questions to transfer:

After use
- Record the outcome and update incorrect instructions.
- Link the incident review and follow-up owner.

Keep commands next to the observations they are meant to produce. If the instructions say to inspect a value, explain which values permit the next step and which require help.

Worked example: restore a worker configuration revision

Suppose a fictional thumbnail service starts delaying jobs after configuration revision r18. Its previous revision, r17, is known to work with the current application and data format. The exercise environment contains synthetic images and a reversible configuration control.

The procedure's objective is to restore useful job completion. Its scope is one named worker group in the test environment. A responder first confirms that the failure began after r18, that the group actually received that revision, and that the compatibility assumption still holds.

Use a decision table to make the sequence reviewable:

Step Expected observation Branch if it differs
Read the worker group's identity and current revision The intended test group reports r18 Stop and resolve the target mismatch
Compare the change record with the last working revision The relevant setting changed between r17 and r18 Return to investigation rather than guessing a revision
Select the documented previous revision The control accepts r17 for this group Escalate with the error and current state
Read back the effective configuration All intended workers report r17 Investigate partial propagation before declaring success
Submit a synthetic job and inspect delayed work New work completes and older valid jobs progress Preserve evidence and reassess the explanation

The labels and revisions are invented. The useful part is the pairing of an action with evidence that determines the next action.

Put the stopping rule beside the risky step

A responder should be able to recognize when the procedure no longer fits. For the example above, an unexpected schema change, a different current revision, or an unaccounted-for job loss would invalidate the simple rollback path.

Place that condition where the decision is made. A warning at the bottom of a long document is easy to miss after someone has already followed the preceding steps.

When the cause is still uncertain, use an investigation workflow to collect evidence and narrow hypotheses. A runbook for a known condition becomes useful after its prerequisites are established. The distributed systems debugging guide provides a structure for that earlier work.

Verify recovery in terms of the service's purpose

A process restart or a successful configuration write establishes that an action occurred. Add checks for what the service is supposed to accomplish.

For the worker example, inspect a new synthetic job, the age of delayed jobs, completion records, and failed-job accounting. Decide how duplicate processing would be detected if the workflow can produce it. Assign someone to watch those checks for an observation window suited to the service.

For changes that affect persisted data, use a database migration rollback plan to establish which earlier versions remain compatible.

Test the document with another responder

Give the runbook to an engineer who did not write it. In the exercise, record each point where they need an unstated hostname, permission, assumption, or expected result. Update the instructions and repeat the affected steps.

Treat a successful walkthrough as evidence tied to the tested environment and revision. Recheck the procedure after a relevant service change or an incident exposes a wrong assumption.

Use the incident communication templates for the handover, and the postmortem template to capture the result. For individual investigation practice before writing a runbook, try a past Incident Challenge.