Cloudflare June 2022 Outage: A Routing Policy Change

Cloudflare | 2022-06-21T00:00:00Z

Cloudflare June 2022 Outage: A Routing Policy Change

Cloudflare's June 21, 2022 outage followed a routing policy change that withdrew critical prefixes in 19 locations using its newer network architecture.

Incident answer

Impact: Users had trouble reaching services through affected Cloudflare locations.

Root cause: Reordered routing policy terms withdrew prefixes required by the affected locations.

Lesson: Choose rollout groups that cover architectural differences and verify a recovery access path.

What caused Cloudflare's June 21, 2022 outage?

Cloudflare's original incident report describes a routing policy change that affected 19 locations using its Multi-Colo PoP architecture. Earlier rollout groups used a different architecture and had not exposed the error. Withdrawing critical prefixes also made recovery access harder.

The incident began at 06:27 UTC. The first location returned at 06:58 UTC, and all affected locations were back by 07:42 UTC. Source: Cloudflare's timeline.

Why an earlier successful rollout may be incomplete evidence

The practical question for your own deployment is which production conditions the first group actually exercises. A small rollout is useful when its observations tell you something about the environments that will receive the change next.

Create a matrix of relevant differences before choosing a rollout order:

Difference Example verification question
Network architecture Does this group exercise each routing layer the policy can affect?
Dependency topology Does it reach the same classes of upstream and downstream systems?
Traffic pattern Does it include the request paths whose behavior changes?
Configuration generation Does it cover every supported configuration variant?
Recovery access Can the operator still reach the management path after the tested failure?

These are suggested checks, not claims about Cloudflare's current deployment system. Choose the differences that matter to the change being reviewed.

Counting healthy machines cannot answer every question in the table. One representative example of a distinct architecture may reveal more than a larger sample of identical machines.

Review policy behavior as well as policy syntax

Ordered rules deserve tests that examine their effect. When rules overlap, moving one can change which outcome applies even when every statement remains valid in isolation.

For a toy routing policy, define sample inputs and expected decisions before changing the order. Include a required route that must remain reachable, a route that must be rejected, and a case that matches more than one rule.

Run the tests against the proposed version and the previous version. Examine the difference in decisions. A clean parser result does not establish that either version implements the intended policy.

The same review habit can help with firewall rules, authorization policies, feature targeting, and request routing. Each has its own semantics, so verify the actual evaluation behavior rather than assuming the first matching rule always wins.

A practice exercise: the canary covers the wrong group

Build a tabletop with two fictional site groups. Group A uses a direct path; Group B uses an extra routing layer. Provide a policy change, successful checks from Group A, and a failed customer-path check from Group B.

Ask participants to:

  1. Identify the architectural difference relevant to the policy.
  2. Explain why Group A's result did not validate Group B.
  3. Predict which policy decision causes the observed failure.
  4. Propose a smaller next rollout that includes the missing architecture.
  5. Describe how they would verify recovery and retain management access.

The exercise can use static decision tables instead of real networking. The learning objective is to connect evidence to the coverage of the rollout, rather than to memorize router commands.

Record a passing result only when the participant can state what the test covers and what remains untested. That is a more useful outcome than calling the first group “the canary” and assuming coverage from the label.

Connect rollout checks to incident practice

Use the team incident drill to practice ownership during a rollback. Use the distributed systems debugging guide to compare affected and unaffected paths.

For a prepared investigation where the final action includes a runtime configuration fix and a deploy, try a past Incident Challenge.

Source and related incidents