Reduce On-Call Alert Fatigue: A Practical Paging Audit

Reduce On-Call Alert Fatigue: A Practical Paging Audit

By The Incident Challenge Team | 2026-11-08T16:21:16Z

To reduce on-call alert fatigue, review which notifications require immediate action, consolidate repeated interruptions, route work to the right owner, and test that important failures still produce a useful page after the changes.

The goal is a responder who can identify the problem and take the next useful step. Notification count is one measurement; the ability to detect and handle significant customer impact is another.

Google's SLO alerting chapter evaluates strategies using precision, recall, detection time, and reset time. Those dimensions help a team assess what it gains or loses when changing a noisy rule.

Audit the interruptions a responder actually receives

Choose a review window that covers representative traffic and operational activity. Include the notifications delivered to people, not only the monitoring system's list of configured rules.

For each interruption, record:

Field Question it should answer
Event and notification identifiers Is this a new problem or another notification about existing work?
Customer operation and scope What outcome may be affected?
Immediate action taken What justified interrupting a person at that moment?
Recipient and service owner Did the page reach someone able to act?
First useful evidence Could the responder find the relevant observation quickly?
Reset and repetition Did notifications continue after recovery or without a new decision?

Preserve links to the incident timeline or ticket so that the review is based on what happened.

A worked audit with fictional data

Suppose a training dataset contains twenty notifications. Six correspond to distinct response tasks requiring immediate action. Nine repeat an already assigned task without new information or a useful escalation. Five concern maintenance or work that the exercise policy allows to wait.

That breakdown suggests different changes. The repeated notifications need grouping or routing review. The five nonurgent items need a delivery path consistent with their urgency. The six immediate tasks need a useful page and a maintained recovery procedure.

Do not use the fictional numbers as a benchmark. A later notification may be useful when ownership was not accepted, impact expands, or escalation becomes necessary. Assess the decision it enables before labeling it a duplicate.

Decide page, ticket, or informational delivery

Write a reason for each destination:

Choose those boundaries using the service's impact and response needs. Record who owns the decision and how exceptions are reviewed.

A high resource measurement may help diagnose a customer problem. To decide whether it warrants a page, ask what the responder should do, how much time is available, and what happens if nobody acts immediately.

Use burn rate with its assumptions visible

For a request-based objective of 99.9% successful requests, the allowed error ratio is 0.1%. If an observation window has a 1% error ratio, the burn rate relative to that objective is ten:

burn rate = observed error ratio / allowed error ratio
          = 1% / 0.1%
          = 10

This is an arithmetic example, not a recommended objective or paging threshold. The ratio does not by itself specify the response deadline. Traffic volume, observation windows, the remaining budget, and the behavior of low-volume services matter when designing the policy.

Make those assumptions part of the alert review. A rule that waits longer may reduce some interruptions while delaying detection. A rule that resets slowly may continue paging after the underlying problem is gone.

Improve the evidence attached to a page

A useful notification identifies the customer operation, affected scope, measurement window, owner, and first investigation link. It should connect to an incident runbook when a known procedure applies.

Include the version or configuration context when it is relevant. A responder should be able to distinguish a new rollout from a dependency problem without reconstructing the service's entire history.

Check the monitoring path itself. Missing observations need a defined interpretation. A graph without new errors can also be a graph without new data. Use the AWS Kinesis case study to start a discussion about monitoring dependencies.

Replay known incidents before accepting the change

Review a proposed rule against recorded or synthetic observations. Include a significant outage, a short transient, a slow degradation, recovery, and a gap in measurements.

For each case, record whether the rule fires, who receives it, how long detection takes, and when it resets. Google's monitoring chapter describes testing monitoring and routing with synthetic data; your own replay should check the behavior your team depends on.

A successful change reduces interruptions that do not enable useful action while preserving the detection and escalation paths the service needs. Assign an owner to evaluate that result after rollout.

Include paging quality in on-call practice

Give a trainee one actionable page and a packet of surrounding noise. Ask them to define impact, identify the next observation, and explain when they would escalate. Then review whether the notification helped or obstructed that decision.

Use the on-call training checklist and SRE training exercises to structure the session. For individual evidence-based debugging practice, try a past Incident Challenge.