GitHub | 2018-02-28T00:00:00Z
GitHub's 2018 DDoS Attack: 1.35 Tbps and a Routing Response
GitHub's February 28, 2018 outage followed a memcached-amplified DDoS attack peaking at 1.35 Tbps. Rerouting traffic through a mitigation provider restored access.
Incident answer
Impact: GitHub.com became unavailable, then intermittently accessible, during the attack.
Root cause: Reflected attack traffic overwhelmed network capacity; this was an availability incident, with no reported data confidentiality or integrity impact.
Lesson: Practice traffic diversion and verify legitimate user access independently of attack-volume graphs.
What happened at GitHub on February 28, 2018?
GitHub's original incident report records unavailability from 17:21 to 17:26 UTC, followed by intermittent access until 17:30 UTC. The team moved traffic through Akamai's mitigation network and used network measurements and load-balancer responses to establish recovery. This was a different event from GitHub's October 2018 MySQL incident.
The engineering review below uses the incident to examine defensive readiness and customer availability.
Separate a traffic anomaly from a customer outage
For your own service, start by asking what each measurement proves. Incoming bandwidth describes demand at one boundary. Accepted connections describe another. A successful authenticated operation describes the customer outcome.
Build a small evidence matrix before choosing a mitigation:
| Observation | Useful question | Additional evidence |
|---|---|---|
| Incoming traffic rises sharply | Is the constrained resource bandwidth, connections, or application work? | Link utilization, connection state, and request completion |
| Origin CPU stays normal | Are requests reaching the application? | Edge responses and external probes |
| Traffic drops after a routing change | Did filtering work, or did legitimate access disappear too? | Independent customer-path checks |
| One region recovers | Do all affected routes reach the intended destination? | Probes from more than one network |
These are proposed investigation questions, rather than a reconstruction of GitHub's internal dashboards. The point is to keep different layers of evidence distinct.
A mitigation path needs its own readiness checks
A traffic-diversion plan is a sequence of dependencies. Someone needs access to the control that initiates it. The destination needs to accept the traffic. Application behavior must survive the new route. Responders need a way to observe the result and decide what happens next.
Walk that sequence before an incident. Record who can make the change, how another responder takes over, and how to recognize an incomplete transition. Review a representative customer operation through the alternative path, including any authentication or routing assumptions it depends on.
A network improvement alone does not answer whether the service is usable. Conversely, one successful probe does not establish that every affected population has recovered. Choose checks that reflect the scope you are trying to restore.
A tabletop exercise: the graph looks better, but access does not
Prepare a fictional service map, two traffic graphs, an access-control change log, and synthetic user checks. No live traffic generation is needed.
In the exercise, a proposed mitigation lowers incoming traffic, but an overly broad rule also blocks one legitimate client population. Give participants the initial traffic anomaly first. Release the customer-path results when they ask how the change affected real operations.
Ask the incident lead to make three decisions:
- Which observations justify invoking the alternative route?
- Which checks distinguish successful filtering from accidental loss of customer access?
- What evidence permits the team to restore normal routing?
The facilitator should accept more than one reasonable response if it explains the available evidence and preserves a safe way to recover. Do not make the exercise depend on remembering a vendor-specific command.
Define completion before making the change
Write a recovery statement with explicit scope: the affected operation completes, the intended customer populations can reach it, and the alternative path remains within its operating limits. Assign an owner to each check.
Preserve the times of detection, mitigation initiation, and verified recovery separately. Those timestamps help a team find delays in access, decision-making, or route convergence without treating them as the same problem.
Use the engineering incident drill plan to rehearse the handover. For another case involving a network change and recovery access, read Cloudflare's June 2022 routing outage.