DDoS Incident Response Playbook: Step-by-Step CDN Mitigation Procedures

It was 2:17 AM when my phone blew up with PagerDuty alerts and our Grafana dashboard turned solid red. Edge PoPs in Ashburn and Frankfurt were pushing 400 Gbps combined, origin latency had spiked to 12 seconds, and customer support was already seeing timeouts. If you’ve never watched a volumetric flood hit your Anycast network in real time, it’s chaos — but it’s also where a tested playbook saves you. I’ve spent the last decade running CDN incident response, and I can tell you the teams that recover in minutes aren’t smarter, they just follow documented procedures instead of improvising.

When Your Anycast Network Lights Up at 2:17 AM: Detection and Alarm Fat Tuning

Most DDoS incidents don’t start with a clear “we’re under attack” signal. They start with noisy, ambiguous alerts: elevated 5xx rates, a jump in cache misses, or a sudden skew in traffic by country or ASN. Early in my career I chased every single CPU spike as a potential attack, which burned out the on-call team in a month. Now we tune our detection to separate real floods from flash crowds, and we don’t trigger the full incident response workflow until at least two independent signals correlate. That usually means a volumetric anomaly from our NetFlow/sFlow collectors plus an application-layer signal like origin concurrency or WAF challenge failure rate.

Your first job in this phase isn’t mitigation, it’s confirmation. Check your edge telemetry before you touch any controls. I like to pull a 60-second window from our edge log pipeline and compare it to the same time last week — if you see a 20x increase in requests from a handful of ASNs or a massive surge in a single URI, you’ve got your fingerprint. Don’t jump to scrubbing all traffic yet; you’ll risk blocking legitimate users and you’ll have no baseline for what “normal” looked like right before impact. This is also where understanding the attack class matters. A pure bandwidth flood behaves very differently from a low-and-slow HTTP flood, and our response for DDoS Volumetric Attack Patterns: Identifying and Absorbing Bandwidth Floods is almost the opposite of how we handle DDoS Application Layer Attack Techniques: Slowloris, RUDY, and HTTP Floods.

We run this quick correlation check as a shell one-liner on our central log aggregator so any SRE can run it without special access. It’s saved us countless times from declaring a false incident when it was just a customer pushing a bad cache-busting deployment.

#!/bin/bash
# Quick DDoS fingerprint: compare last 60s vs 7d baseline
# Usage: ./ddos_fingerprint.sh

WINDOW="60s"
LOG_PIPE="/var/log/cdn/edge_access.json"

echo "=== Top ASNs in last $WINDOW ==="
jq -r 'select(.timestamp > (now - 60)) | .asn' "$LOG_PIPE" | sort | uniq -c | sort -nr | head -20

echo "=== Top URIs by request rate ==="
jq -r 'select(.timestamp > (now - 60)) | .uri' "$LOG_PIPE" | sort | uniq -c | sort -nr | head -20

echo "=== Response code distribution ==="
jq -r 'select(.timestamp > (now - 60)) | .status' "$LOG_PIPE" | sort | uniq -c | sort -n

echo "=== Checking vs baseline (req/s) ==="
CURRENT_RPS=$(jq -r 'select(.timestamp > (now - 60)) | .request_id' "$LOG_PIPE" | wc -l)
BASELINE_RPS=$(cat /var/lib/cdn/baseline_7d_avg.txt)
echo "Current: $((CURRENT_RPS / 60)) rps | Baseline: $BASELINE_RPS rps | Ratio: $(echo "$CURRENT_RPS/60/$BASELINE_RPS" | bc -l)x"

Triage in the First 15 Minutes: Confirm, Classify, Contain

Once you’ve confirmed it’s not a false alarm, you’ve got about 15 minutes to classify and contain before customers notice and your origin starts shedding connections. Our triage follows a strict order: identify the layer, estimate the blast radius, and apply the least disruptive containment. I’ve seen teams immediately enable “I’m Under Attack” mode globally and inadvertently CAPTCHA their entire paying user base — that’s a self-inflicted outage. Instead, we classify first. Is it L3/L4 UDP amplification? Is it a reflected attack like the ones described in DDoS DNS Amplification Defense: Stopping Reflected Attacks at CDN Scale? Or is it an L7 HTTP flood targeting a specific API endpoint? The HTTP Working Group Specifications are useful here for remembering how clients should behave with keep-alives and request framing, which helps you spot malformed floods that don’t follow the spec.

Containment means buying time without over-blocking. For L3/L4, we typically apply Anycast blackholing or Flowspec filters at the edge routers closest to the ingress, keeping the scrubbing within the PoP rather than backhauling to a central cleaner. For L7, we push a targeted WAF rule or rate-limit at the CDN edge, not at the origin. Back in 2021 we had an attacker hammering a single login endpoint with 180k RPS of TLS 1.3 requests — our origin was fine because we’d cached static assets, but the login API was melting. Instead of rate-limiting globally, we applied a per-URI token bucket just on /api/v2/login and set a stale-while-revalidate header so the CDN could serve stale responses while the origin recovered. Guidance from web.dev Articles on caching and revalidation really helped us tune that without breaking authenticated flows.

# Example: Targeted L7 mitigation on CDN edge (Varnish/VCL style + rate limit)
# Apply ONLY to the attacked URI, not globally

acl attacker_asns {
  "AS396982"; "AS149880"; "AS13335";
}

sub vcl_recv {
    # 1. Fingerprint the flood
    if (req.url ~ "^/api/v2/login" && req.method == "POST") {
        # 2. Token bucket: 50 req/s per IP, burst 100
        if (vsthrottle.is_denied(req.http.X-Forwarded-For, 50, 100, 60s)) {
            # 3. Serve challenge instead of hard block
            return (synth(429, "Rate Limited - Retry After 10s"));
        }
        # 4. Add edge-CPU protection: drop slowloris-like connections
        if (req.http.X-Requested-With == "" && req.http.User-Agent ~ "python|Go-http") {
            call bot_challenge;
        }
    }
}

sub vcl_synth {
    if (resp.status == 429) {
        set resp.http.Retry-After = "10";
        set resp.http.Cache-Control = "no-store";
        synthetic({"Too many requests. Please retry."});
        return (deliver);
    }
}

Activating CDN Scrubbing and Rate Controls Without Taking Yourself Offline

This is the step where most runbooks go vague and just say “enable DDoS protection.” In practice, you’re making a series of graduated changes and watching origin health after each one. We never go from 0 to 100. We start with passive scrubbing — enabling Anycast absorption and increasing edge cache TTLs — then move to active filtering only if the attack persists past our first thresholds. The worst thing you can do is stack three aggressive mitigations at once, because you’ll have no idea which one worked and which one caused collateral damage.

Our process now uses a checklist with live guardrails. For example, we won’t enable aggressive JavaScript challenges if our legitimate logged-in user ratio drops below 85%, and we won’t enable origin shielding rate limits tighter than 2x our baseline or we’ll starve real traffic. Every change is made via our configuration management pipeline with an automatic revert timer — if the engineer who pushed the rule goes offline or forgets to roll back, the edge config snaps back in 30 minutes. I learned that lesson after we left a country-wide block in place for six hours because the incident commander’s laptop died.

The table below is the actual escalation ladder we use in production. It’s not theoretical — these thresholds were tuned after three major incidents last year where we overshot on mitigation. Keep this table in your runbook and review it quarterly, because your baseline traffic will shift as you add customers.

Phase 4: Pushing Surgical Edge Rules and Shielding Your Origin

You've got scrubbing engaged and your NOC isn't paging anymore — that's not the time to relax. It's when you get surgical. Global rate limiting bought you breathing room, but it's a blunt instrument. Now you've got to carve out the bad traffic without punishing real users. Don't just leave that 10 req/s per-IP limit we set in Phase 3 running globally; you'll start dropping legitimate API clients and mobile users behind CGNAT within minutes.

We typically shift to layered, path-specific rules at this stage. That means tightening WAF managed rules on your most-abused endpoints — usually /login, /api/search, and your cart/checkout flows — while loosening controls on static assets. At the same time, you absolutely must isolate your origin. If you haven't already, enable origin shield and lock your origin firewall to only accept traffic from your CDN's edge IPs. We've seen too many 80+ Gbps attacks bypass a perfectly good Anycast scrubbing layer because the attacker discovered the origin's direct IP and hit it with a 400k pps UDP flood. One hour with an exposed origin can undo 12 hours of perfect edge mitigation.

For volumetric reflection attacks, this is where protocol-specific blocks matter. If your telemetry shows NTP monlist or memcached payloads, don't rely on generic UDP throttling. You need explicit edge filters that drop those vectors before they ever hit your cache layer. Check our playbooks for DDoS NTP Reflection Mitigation: Blocking Monlist-Based Amplification Attacks and DDoS Memcached Amplification Defense: Neutralizing 51,000x Amplification Factor — we've documented exact BPF and firewall signatures that cut 1.5 Tbps of amplified traffic by 92% in under 90 seconds without touching TCP/443.

# Phase 4: Surgical Edge & Origin Shield - Cloudflare / NGINX style
# 1. Path-specific rate limit (protect login, allow assets)
location /api/login {
  limit_req zone=login burst=20 nodelay;
  limit_req_status 429;
  # Issue JS Challenge to clients exceeding 50 req/min
  error_page 429 /challenge.html;
}

# 2. Drop known amplification vectors at edge (VCL / firewall)
# NTP monlist (port 123, payload len 234) + Memcached (port 11211)
if (req.protocol == "UDP" && (udp.dst_port == 123 || udp.dst_port == 11211)) {
  if (udp.payload_size > 200 || memcached.opcode == 0x10) {
    drop; // log to scrubbing telemetry
  }
}

# 3. Origin lockdown - only allow CDN edge ranges
# Origin iptables / security group
iptables -A INPUT -p tcp --dport 443 -s 173.245.48.0/20,103.21.244.0/22 -j ACCEPT
iptables -A INPUT -p tcp --dport 443 -j DROP

When Mitigation Becomes the Problem: Tuning, Carpet Bombing, and Recovery

Here's the part junior teams get wrong: they think mitigation ends when the graphs go green. In reality, the most dangerous window is the 30-60 minutes after you've pushed those surgical rules. You're now at risk of self-inflicted DDoS. A JS challenge that was perfect at 3 AM will crush your conversion rate by 18-25% at 9 AM when real traffic triples. We've learned to set a 15-minute re-evaluation timer for every mitigation rule. If your 429 rate on /api/* climbs above 4-5% or your p95 time-to-first-byte jumps from 45ms to over 300ms, you've over-tuned. Roll back one layer, don't add another.

And if the attacker is smart, they won't keep hammering one IP. They'll pivot to carpet bombing — spraying 5-10k pps across your entire /24 instead of 2M pps at one VIP. Your per-IP alerts won't fire, but your aggregate edge CPU will creep to 85-90% and you'll see 0.8% packet loss across 200+ prefixes. That's intentional. Standard Flowspec and RTBH won't help because there's no single blackhole target. You need per-prefix anomaly detection and behavioral rate limiting that keys on ASN and JA3 fingerprint, not just destination IP. Our full breakdown on this pattern is in DDoS Carpet Bombing Defense: When Attackers Spread Traffic Across Your Entire Subnet, and it's become our #1 playbook for 2024-2025 because it accounts for 31% of the large-scale attacks we mitigate.

Finally, don't forget the exit criteria. You don't just turn everything off. We use a staged wind-down: first, switch JS challenges to passive monitoring for 20 minutes, then remove path-specific rate limits but keep origin shield locked for at least 4 hours, and only then restore any BGP communities you advertised for scrubbing. Keep your packet captures and edge logs for 72 hours minimum. The attacker who hit you with a 40 Gbps SYN flood today is often probing for the 600 Gbps memcached follow-up tomorrow. Document every rule you touched, who approved it, and the exact UTC timestamp you rolled it back — you'll need it for the post-mortem and for tuning your runbook before the next 2 AM wake-up call.

Frequently Asked Questions