Spotify | 2022-03-08T00:00:00Z
Spotify's March 2022 Outage: A Service Discovery Failure
Spotify's March 8, 2022 login outage combined a Google Cloud Traffic Director failure with a gRPC client-library bug. Moving affected systems back to DNS-based discovery restored service.
Incident answer
Impact: Affected users could not log back into Spotify applications.
Root cause: Traffic Director disruption and client-library behavior broke discovery for affected services; the provider incident involved an incomplete configuration migration.
Lesson: Verify that fallback discovery works end to end and has an activation path independent of the failed control plane.
Why could Spotify users not log in on March 8, 2022?
Spotify's engineering report attributes the outage to Google Cloud Traffic Director disruption interacting with a gRPC client-library bug. Spotify moved affected systems back to DNS-based service discovery and recovered gradually.
Spotify published this response timeline:
| Time, UTC | Event |
|---|---|
| 18:12 | Reports of users being logged out began surfacing |
| 18:39 | Remediations were being put in place |
| 20:35 | The incident was fully mitigated at Spotify |
These are Spotify's milestones, rather than the provider's incident window.
The provider's configuration migration was incomplete
Google Cloud's separate incident report describes a Traffic Director change that assumed a configuration-format migration had completed. Some customer configurations still mixed old and new formats. Processing that state could delete configurations and leave clients without their programming. Google rolled back the change and forced configuration reprogramming.
That account explains the provider failure. Spotify's report supplies the customer-service impact and its own mitigation. Keeping both views prevents a provider's restoration time from being mistaken for proof that every dependent customer workflow recovered.
A fallback is a complete request path
For your own service, draw the primary and alternative discovery paths from the caller's point of view. Include the configuration that selects the path, the addresses it returns, and the connection that ultimately reaches a healthy backend.
Then inspect the shared dependencies. Two discovery methods can still depend on the same credential, deployment mechanism, network route, or responder access. If that common dependency fails, a second implementation may not provide an independent recovery option.
A useful readiness review asks:
- Can the responder activate the alternative while the primary control plane is unavailable?
- Does the caller actually use the changed setting, or retain an old connection or cached result?
- Are returned endpoints reachable with the required credentials and routing rules?
- Does the alternative preserve the application's expected behavior?
- What observation tells the team that the transition has completed?
These are proposed checks for your environment, not a description of Spotify's current implementation.
Test warm connections and fresh starts separately
An already connected caller and a newly started caller can encounter different failure paths. One may continue using a known destination while the other needs a fresh discovery result.
Design a test that observes both. Record which information the caller already possesses, what must be fetched again, and when the application adopts a configuration change. Avoid treating a single healthy long-lived connection as evidence that a new instance can recover.
The same distinction matters when returning to the preferred path. Define the conditions for that transition and watch the customer operation during it, rather than using the provider's status update as the only signal.
A practice exercise: the switch changed, but the caller did not
Create a tabletop or an isolated toy service with primary and fallback endpoint lists. Provide two synthetic clients: one keeps an established connection and one starts fresh. Give the facilitator a controlled discovery failure and a known recovery configuration.
Ask participants to compare results before and after the switch. Include the configuration revision seen by each client, the selected endpoint, connection attempts, and completed customer operations.
Success requires an explanation of which path each client is using and evidence that both can complete the intended operation. A successful configuration write alone should not pass the exercise. Repeat the check when switching back.
Use the distributed systems debugging guide to structure the observations. Compare this case with Roblox's 2021 Consul outage, another incident in which a shared infrastructure dependency affected recovery.