Spotify's March 2022 Outage: A Service Discovery Failure

Spotify | 2022-03-08T00:00:00Z

Spotify's March 2022 Outage: A Service Discovery Failure

Spotify's March 8, 2022 login outage combined a Google Cloud Traffic Director failure with a gRPC client-library bug. Moving affected systems back to DNS-based discovery restored service.

Incident answer

Impact: Affected users could not log back into Spotify applications.

Root cause: Traffic Director disruption and client-library behavior broke discovery for affected services; the provider incident involved an incomplete configuration migration.

Lesson: Verify that fallback discovery works end to end and has an activation path independent of the failed control plane.

Why could Spotify users not log in on March 8, 2022?

Spotify's engineering report attributes the outage to Google Cloud Traffic Director disruption interacting with a gRPC client-library bug. Spotify moved affected systems back to DNS-based service discovery and recovered gradually.

Spotify published this response timeline:

Time, UTC Event
18:12 Reports of users being logged out began surfacing
18:39 Remediations were being put in place
20:35 The incident was fully mitigated at Spotify

These are Spotify's milestones, rather than the provider's incident window.

The provider's configuration migration was incomplete

Google Cloud's separate incident report describes a Traffic Director change that assumed a configuration-format migration had completed. Some customer configurations still mixed old and new formats. Processing that state could delete configurations and leave clients without their programming. Google rolled back the change and forced configuration reprogramming.

That account explains the provider failure. Spotify's report supplies the customer-service impact and its own mitigation. Keeping both views prevents a provider's restoration time from being mistaken for proof that every dependent customer workflow recovered.

A fallback is a complete request path

For your own service, draw the primary and alternative discovery paths from the caller's point of view. Include the configuration that selects the path, the addresses it returns, and the connection that ultimately reaches a healthy backend.

Then inspect the shared dependencies. Two discovery methods can still depend on the same credential, deployment mechanism, network route, or responder access. If that common dependency fails, a second implementation may not provide an independent recovery option.

A useful readiness review asks:

These are proposed checks for your environment, not a description of Spotify's current implementation.

Test warm connections and fresh starts separately

An already connected caller and a newly started caller can encounter different failure paths. One may continue using a known destination while the other needs a fresh discovery result.

Design a test that observes both. Record which information the caller already possesses, what must be fetched again, and when the application adopts a configuration change. Avoid treating a single healthy long-lived connection as evidence that a new instance can recover.

The same distinction matters when returning to the preferred path. Define the conditions for that transition and watch the customer operation during it, rather than using the provider's status update as the only signal.

A practice exercise: the switch changed, but the caller did not

Create a tabletop or an isolated toy service with primary and fallback endpoint lists. Provide two synthetic clients: one keeps an established connection and one starts fresh. Give the facilitator a controlled discovery failure and a known recovery configuration.

Ask participants to compare results before and after the switch. Include the configuration revision seen by each client, the selected endpoint, connection attempts, and completed customer operations.

Success requires an explanation of which path each client is using and evidence that both can complete the intended operation. A successful configuration write alone should not pass the exercise. Repeat the check when switching back.

Use the distributed systems debugging guide to structure the observations. Compare this case with Roblox's 2021 Consul outage, another incident in which a shared infrastructure dependency affected recovery.

Sources and further practice