Google's 2020 Authentication Outage: A Quota Failure

Google | 2020-12-14T00:00:00Z

Google's 2020 Authentication Outage: A Quota Failure

Incorrect usage reporting during a quota-system migration reduced capacity for Google's User ID database. Authentication lookups then failed during the December 14, 2020 outage.

Incident answer

Impact: Services requiring Google OAuth access were unavailable during the core 47-minute incident; cloud service accounts continued to function.

Root cause: After a grace period expired, quota automation acted on falsely reported zero usage, blocking database updates and making identity data stale.

Lesson: Test delayed policy activation and zero-usage inputs, and preserve authentication checks while restoring capacity.

What caused Google's December 2020 authentication outage?

Google's final incident report traces the failure to quota enforcement against account storage. The identity service rejected outdated data rather than trusting stale authentication lookups. Engineers disabled quota enforcement, first in one datacenter and then more broadly, to restore the service.

The review questions below apply that failure pattern to your own systems. They are not claims about Google's current infrastructure.

Missing data and zero usage require different decisions

Suppose a fictional policy engine reads daily usage from a metrics feed. The feed has no fresh observation for one service. Should the engine treat that as zero demand, retain the previous allocation, or stop the proposed change for review?

The answer depends on the contract, but it must be explicit. A number alone cannot tell a consumer whether it represents a measurement, a default, an error, or an expired observation.

For each input that can reduce capacity, record:

A schema can validate that a field is numeric while still allowing a dangerous interpretation. Review the meaning of the input as well as its type.

A grace period is another execution path

A change can be dormant until a timer, lease, certificate, compatibility window, or policy exemption expires. Reviewing only the immediate rollout leaves that later transition untested.

Build an activation timeline for your own migration. Include the new registration, the old system's retirement, the expiration of any temporary exception, and the point where enforcement starts using the new interpretation.

Ask which component owns each transition and which observation proves it happened. If two systems can still produce policy inputs, define how disagreement is resolved. Do not assume that an uneventful waiting period validates the behavior that follows it.

A tabletop exercise: the deployment was last month

Prepare a fictional identity service, a storage quota policy, an old usage feed, and a migration record. At the beginning of the exercise, a previously approved grace period expires. Fresh account updates stop, and authenticated requests fail.

Give participants an unchanged application deployment history for the current day. Provide the older migration record when they investigate policy history. Include a read-only customer path that is unaffected, so they can compare the scope.

The participant should answer three questions:

Question Evidence to seek
What activated now? Policy evaluation time and expiration record
Why did the policy choose this value? Input provenance, freshness, and interpretation
What restores service without weakening identity checks? A capacity or policy correction and successful authenticated operations

This is an original teaching scenario. It does not reproduce Google's architecture or prescribe an emergency change to a real authentication system.

Restore availability without confusing it with authorization

For your own recovery plan, distinguish a healthy identity lookup from a bypass of the lookup. A successful response is insufficient if the operation no longer enforces the intended access rule.

Use both positive and negative verification: an authorized synthetic account should complete its operation, and an account without permission should remain denied. Check whether account changes become visible within the system's required freshness window.

Include the operational path too. Decide how responders will reach the controls and communications they need if the ordinary sign-in path is unavailable.

Use the on-call training checklist to rehearse those access assumptions. For another case involving automation and unexpected input, read Google Cloud's June 2025 Service Control outage.

Source and further practice