DDoS Scrubbing Center Architecture: Designing Clean-Pipe Infrastructure

At 2:14 a.m. last November I watched 1.2 Tbps of junk UDP hit our Frankfurt PoP before the upstream NOC even paged us. Our automated diversion had already shifted the victim /24 to the scrubbing center, and customer traffic barely hiccuped. If your clean pipe can't make that decision in under 30 seconds without human intervention, you don't have DDoS mitigation infrastructure — you've got a very expensive transit bill. That's the architecture we're going to build here.

How Clean-Pipe Actually Works: Diversion Before Destruction

Forget the marketing diagram where traffic magically goes through a cloud. A real scrubbing center is a parallel network you intentionally steer traffic into, clean, and then re-inject. The core idea behind a clean pipe is simple but brutal to get right: you maintain pristine capacity that never sees dirty traffic until an attack is confirmed, and you keep the decision-to-diversion loop as short as possible. I've seen teams try to run scrubbing inline on every edge router. It works until a 800 Gbps DDoS Volumetric Attack Patterns: Identifying and Absorbing Bandwidth Floods saturates your backbone before any filter can fire.

In our current design, detection lives at the edge, but scrubbing doesn't. We use sFlow and IPFIX sampled at every border router, feeding a central detector cluster that runs both threshold and behavioral models. When it flags a destination — say, >5x baseline PPS or a sudden entropy shift in source ports — it triggers BGP diversion. We've learned not to rely on a single signal. Back in 2019 we had a false diversion that blackholed a customer's legitimate flash sale because we triggered on bandwidth alone. Now we correlate flow data, SNMP interface counters, and SYN backlog on the origin before we move a prefix.

Once flagged, we announce a more-specific /24 (or /48 for IPv6) with a scrubbing community, typically 64999:0, to our transit and peer routers. Upstreams honor it and hairpin traffic to our scrubbing PoPs via GRE or MPLS tunnels. Clean traffic returns via separate VRFs or L2VPN back to the origin data center. That separation is critical — you can't return scrubbed traffic on the same path you dragged the attack in on, or you'll congest your own return. The HTTP Working Group Specifications are useful context here for understanding why we can't just drop and reassemble at layer 7 without maintaining strict protocol semantics for the clean traffic.

# BGP diversion announcement for scrubbing center - FRANKFURT-SCRUB-01
# Triggered automatically by detector, withdrawn on mitigation end

route-map SCRUB-DIVERSION permit 10
 match ip address prefix-list VICTIM_PREFIX
 set community 64999:0 65001:100 no-export
 set ip next-hop 192.0.2.1  # scrubbing center GRE endpoint
 set local-preference 200

# Flowspec rule to drop reflected NTP monlist at the edge BEFORE diversion
flow route 65001:1
 match destination 203.0.113.0/24
  protocol udp
  destination-port 123
  packet-length 468
 then discard

Inside the Scrubbing Stack: Where Traffic Actually Gets Cleaned

Diversion without effective traffic scrubbing is just moving the problem. Once traffic lands in the scrubbing center, it hits a tiered pipeline and the order matters. Tier 1 is always stateless and runs in hardware or XDP/eBPF — we're talking ACLs, BGP Flowspec, and port-based drops that can handle tens of millions of PPS per box without touching conntrack. This is where you kill the obvious garbage: UDP amplification payloads, fragmented packets with bad checksums, and sources from your permanent blocklist. If you try to do stateful inspection here, your scrubbing nodes will melt. Don't.

Tier 2 is where we spend most of our engineering time. This is stateful scrubbing: SYN cookies, rate limiting per 5-tuple, challenge-response for suspicious HTTP, and behavioral fingerprinting. For volumetric floods it's straightforward, but for sneaky stuff like DDoS Application Layer Attack Techniques: Slowloris, RUDY, and HTTP Floods you need L7 awareness. We had an attack last spring that was just 40k RPS of perfectly valid-looking POSTs to a login endpoint — it never tripped our PPS thresholds. Our eBPF SYN proxy let it right through because the TCP handshake was clean. We only caught it because our L7 tier tracked request entropy and cookie reuse per IP. You can't scrub what you don't parse, and guidance on how browsers and CDNs should handle connection limits in web.dev Articles actually helped us tune our thresholds so we didn't nuke real users.

Tier 3 is the safety net: per-customer BGP Flowspec rules, per-URL rate limits, and origin protection. We keep this configurable via API so customers can push their own rules during an attack without calling us. The biggest mistake I've seen is building a scrubbing center that only the network team can operate. If your SOC can't push a filter in 60 seconds at 3 a.m., you've failed. Our pipeline exposes a simple abstraction — drop, rate-limit, challenge — and compiles it down to nftables, XDP, and Envoy filters automatically.

// XDP stateless filter - Tier 1 scrubbing node
// Drops NTP amplification and DNS reflection before conntrack
SEC("xdp/scrub_tier1")
int scrub_filter(struct xdp_md *ctx) {
    void *data_end = (void *)(long)ctx->data_end;
    void *data = (void *)(long)ctx->data;
    struct ethhdr *eth = data;
    struct iphdr *iph;
    struct udphdr *udph;

    if (eth + 1 > data_end) return XDP_PASS;
    if (eth->h_proto != htons(ETH_P_IP)) return XDP_PASS;

    iph = data + sizeof(*eth);
    if (iph + 1 > data_end) return XDP_PASS;
    if (iph->protocol != IPPROTO_UDP) return XDP_PASS;

    udph = (void *)iph + sizeof(*iph);
    if (udph + 1 > data_end) return XDP_PASS;

    // NTP monlist - 468 bytes, a classic in DDoS DNS amplification defense
    if (ntohs(udph->dest) == 123 && ntohs(udph->len) == 468) {
        return XDP_DROP;
    }
    // DNS amplification - large responses, small queries
    if (ntohs(udph->source) == 53 && ntohs(udph->len) > 512) {
        // Sample 1:100 for analysis, drop rest if under attack flag
        if (bpf_map_lookup_elem(&attack_active, &key))
            return XDP_DROP;
    }
    return XDP_PASS;
}

Sizing a Scrubbing Center That Won't Collapse Under Real Load

Everyone overestimates their filtering capacity and underestimates their return path. We design scrubbing centers for 60% clean throughput under full attack load, not 100%. Why? Because when you're scrubbing 2 Tbps down to 200 Gbps of legitimate traffic, you still need to deliver that 200 Gbps back to the origin without dropping it. I've audited centers that could ingest 3 Tbps but only had 400 Gbps of clean egress — they scrubbed successfully and then blackholed their own customers on the way out. Your DDoS mitigation infrastructure has to be sized end-to-end.

The numbers below are from our EU cluster after we rebuilt it in 2023. We run three scrubbing PoPs, each with identical hardware, and we anycast the scrubbing prefix so diversion automatically load-balances. We also keep one PoP entirely idle as N+1 — it never serves production traffic until failover. That idle capacity saved us when two PoPs were hit simultaneously by a carpet-bombing attack that targeted every /24 in a /16. If you're planning for single-prefix attacks only, you're planning for 2015. Modern campaigns, especially reflected attacks detailed in DDoS DNS Amplification Defense: Stopping Reflected Attacks at CDN Scale, will spray across your entire address space.

Don't forget state table exhaustion. A SYN flood at 20M PPS won't saturate your link, but it'll wipe out a stateful firewall's conntrack in seconds. We moved all Tier 1 to stateless XDP for that reason and keep connection tracking only in Tier 2 with strict timeouts — 30 seconds for half-open, 10 seconds for unverified HTTP. Test this with real traffic replay, not synthetic iperf. Replay your top customer's busiest hour through the scrubbing stack and see where it chokes.

Implementing Diversion and Reinjection Without Blackholing Yourself

Design diagrams look clean. Production BGP doesn't. The difference between a scrubbing center that works and one that takes you offline is how tightly you control diversion and reinjection. We don't use manual route injections — that's 2015 thinking. Our detection layer (sFlow/IPFIX + in-line telemetry from edge routers) triggers diversion automatically when traffic to a /32 exceeds baseline by 3x for more than 15 seconds or hits 40 Gbps. That signal goes to an ExaBGP controller that advertises a more-specific /32 with community 64999:0 to our transit providers, steering just the victim prefix to the scrubbing PoP while the rest of your /24 stays on the clean path.

You'll want two things nailed before you advertise anything: clean traffic return and MTU handling. We reinject scrubbed traffic via GRE over dedicated L2 circuits — not over the public internet — with at least 20% headroom over your contracted clean throughput. If you've got a 400 Gbps scrubbing capacity, provision 480 Gbps of reinjection or you'll drop clean traffic right after you've paid to clean it. And don't forget GRE overhead. We clamp MSS to 1380 and set the tunnel MTU to 1476. Forget that and you'll see 3-5% packet loss on large POSTs that looks exactly like the attack's still ongoing. Asymmetry's another killer — make sure your reinjection point is topologically close to the diversion point or you'll create 40-60ms of added latency that'll timeout your API clients.

# ExaBGP controller - automated diversion + Flowspec for reflection
# Announce victim /32 to scrubbing center (AS 64500)

neighbor 192.0.2.1 {
  router-id 192.0.2.10;
  local-as 64512;
  peer-as 64500;
  static {
    # Diversion: /32 with scrubbing community
    route 203.0.113.45/32 next-hop 198.51.100.1 community [64999:0, 64500:100] {
      # NO_EXPORT to keep diversion inside scrubbing fabric
    }
    # Flowspec: drop NTP monlist and memcached reflection on the way in
    flow route 203.0.113.45/32 {
      match {
        source 0.0.0.0/0;
        destination 203.0.113.45/32;
        protocol udp;
        source-port 123;
      }
      then {
        discard;
      }
    }
  }
}

# Reinjection - Linux GRE (on scrubbing egress)
# ip tunnel add gre-clean mode gre remote 203.0.113.1 local 198.51.100.2 ttl 255
# ip link set gre-clean mtu 1476 up
# iptables -t mangle -A POSTROUTING -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

Where Clean-Pipe Fails: Evasion Patterns and Troubleshooting at Scale

The classic volumetric attack is easy — 1.2 Tbps of UDP, you scrub it, you win. What's actually collapsing scrubbing centers right now is DDoS Carpet Bombing Defense: When Attackers Spread Traffic Across Your Entire Subnet paired with pulse-wave tactics. Instead of hammering one IP with 800 Gbps, attackers spray 50k pps to every IP in your /22 or /20. Your per-IP thresholds never fire, but your edge routers are suddenly handling 900 Mpps aggregate. If your detection only watches /32s, you won't divert anything until customers are already timing out. We've had to move to subnet-level baselining — tracking the 95th percentile for your entire advertisement and triggering diversion on a 2.5x subnet deviation, even if no single IP looks anomalous. It's noisier, but it catches carpet bombing 30-40 seconds earlier.

Then there's the reflection layer that gets past your L3 scrub. NTP and memcached aren't dead, they've just gotten smarter. We still see 30-50 Gbps NTP monlist floods weekly, and a single memcached amplifier can still push that brutal 51,000x factor if you haven't filtered it upstream. You can't just rate-limit UDP/123 or UDP/11211 and call it done — you'll block legitimate telemetry and caching traffic. The right approach is stateless filtering at the scrubbing ingress *before* it hits your stateful engines. Drop NTP mode 7 packets and memcached UDP with payload size > 1,400 bytes at the edge ACL, then let SYN proxy and challenge handle the rest. We've documented the exact filters we use in our guides on DDoS NTP Reflection Mitigation: Blocking Monlist-Based Amplification Attacks and DDoS Memcached Amplification Defense: Neutralizing 51,000x Amplification Factor — copy those BPFs instead of reinventing them.

When troubleshooting, don't stare at throughput graphs. Look at state and pps. Most scrubbing center failures I've seen weren't bandwidth exhaustion — they were conntrack or SYN proxy table exhaustion at 60-70% of advertised capacity. A 500 Gbps scrubber rated for 150 Mpps will fall over at 80 Mpps of 64-byte SYN flood if your challenge table is set to 4M entries. Monitor your scrubbing stack's session table utilization, not just link utilization, and set alerts at 65%. If reinjected traffic shows 0.8% retransmits and your scrubber CPU is at 85%, you're not under-provisioned — you've got a state leak. Flush, tune your idle timeout from 300s down to 30s for challenged flows, and turn on SYN cookies earlier than you think you need to.

Frequently Asked Questions