DNS Sinkholing at CDN Scale: Redirecting Malicious Traffic for Analysis

Back in late 2021, we watched a Mirai variant hammer one of our customer domains with over 400,000 queries per second — all randomized subdomains trying to bypass cache. Our authoritative DNS was holding, but the origin couldn't. We didn't block it outright. Instead, we flipped that domain's malicious pattern into a sinkhole, routed the junk traffic to a scrubbing cluster we control, and spent the next 48 hours dissecting its behavior without impacting a single legitimate user. That's the quiet power of DNS sinkholing done right.

How Sinkholing Actually Works Inside an Authoritative CDN DNS Stack

Most folks think of a DNS sinkhole as a simple trick — just point a bad domain to 127.0.0.1 and call it a day. At CDN scale, that approach falls apart fast. You're not running a Pi-hole for your house; you're running a globally distributed, anycasted authoritative DNS platform that answers 20+ million queries per second. You can't just edit a zone file. You need a policy layer that sits between the query and the answer, evaluates reputation in real time, and decides to return a controlled, analyzable response instead of the real one. For us, that layer is Response Policy Zone (RPZ) logic baked directly into our edge DNS software, fed by a threat intelligence pipeline that updates every 30 seconds.

I've seen teams try to do this with standard DNS firewall rules and quickly drown in operational overhead. The key isn't blocking — it's redirecting with intent. When we flag a domain or even a specific fully qualified domain name pattern as malicious, we don't return NXDOMAIN. We return a synthetic A or AAAA record that points to our sinkhole infrastructure. That infrastructure is itself a set of hardened, anycasted nodes designed to complete the TCP handshake, capture HTTP payloads, TLS fingerprints, and JA3 hashes, and then drop the connection gracefully. This lets us do DNS NS Delegation for CDN: Offloading Domain Resolution to Edge Networks safely while still keeping visibility into what the botnet *thinks* it's doing.

One gotcha we learned early on: you have to handle EDNS Client Subnet for CDN Geolocation: Routing Users to the Nearest Edge correctly. If you sinkhole without preserving or properly anonymizing the ECS data, you lose the ability to geolocate the source of malicious traffic, which is half the reason you're sinkholing in the first place. We also had to be careful with DNS CNAME Flattening: Pointing Root Domains to CDN Without Breaking DNS Rules — if a sinkholed record is involved in a flattening chain at the apex, you must interrupt the chain before resolution, not after. Our implementation checks the RPZ verdict before any flattening or alias resolution kicks in. For anyone building this, the HTTP Working Group Specifications are essential reading on how clients will retry and behave when they get that sinkholed response versus a hard failure.

# PowerDNS-style RPZ / Lua policy for selective sinkholing
-- Policy feed is pulled from our threat intel API every 30s
rpzMaster("threat-feed.internal", "rpz.malicious.example.com", {
    refresh = 30,
    maxTTL = 60,
    action = "custom"
})

function preresolve(dq)
    -- dq.qname is the query, dq.remoteaddr is client IP (with ECS if present)
    verdict, sinkhole_ip = checkThreatIntel(dq.qname:toString())
    if verdict == "malicious" then
        -- Don't return NXDOMAIN; return our analysis cluster
        -- Preserve ECS subnet for geo-analysis, log original qname
        dq:addAnswer(pdns.A, sinkhole_ip) -- e.g., 192.0.2.50 (RFC5737 sinkhole range)
        dq:addAnswer(pdns.TXT, "sinkhole-id=" .. logToKafka(dq))
        logSinkholeEvent(dq.qname, dq.remoteaddr, dq.ecs)
        return true -- stop further processing
    end
    if verdict == "suspicious" then
        -- For grey-listed domains, add a short TTL to allow quick reversal
        dq.variableTTL = 30
    end
    return false -- continue normal resolution
end

Selective Redirection Without Breaking Legitimate Traffic

The hardest part of threat mitigation at this scale isn't catching the bad stuff — it's not catching the good stuff. A false positive on a sinkhole rule for a popular domain is a self-inflicted outage. We learned that the hard way in 2019 when an overly broad regex for a DGA (Domain Generation Algorithm) flagged a legitimate customer subdomain that happened to look random. It took us 12 minutes to roll back, but that felt like an eternity. Since then, we've moved to a three-tier verdict system: confirmed malicious, high-confidence suspicious, and low-confidence anomalous. Only the first tier gets an automatic hard sinkhole. The rest go through sampling, client reputation scoring, and rate analysis.

This is where you have to treat DNS security as a data problem, not just a networking one. We don't sinkhole based on a domain name alone. We correlate the query name, query type, query rate per /24, ECS-derived ASN, and historical NXDOMAIN rate for that client. If a single IP in Brazil suddenly asks for 5,000 algorithmically generated subdomains of a banking site under our protection, that's a strong sinkhole candidate. If a thousand diverse residential IPs ask for the same new subdomain once, that's probably a marketing campaign, not malware. Our edge Lua scripts can make that decision in under 0.3ms, which matters when you're doing it millions of times a second. Guidance on modern client behavior and retry storms after a sinkhole response is well documented in web.dev Articles, and it’s helped us tune our TTLs and TCP handling on the sinkhole side.

We also maintain a strict allowlist override path that propagates faster than the blocklist itself. If our SOC or a customer flags a false positive, that exception is pushed via our control plane to all 250+ PoPs in under 5 seconds, with a lower priority than any manual block but higher than the automated feed. And we never sinkhole the root domain itself if we can identify a specific malicious FQDN pattern — for example, we’ll sinkhole *.malware.example.com but leave example.com and www.example.com untouched. That granularity is what separates a useful DNS security tool from a blunt instrument.

// Edge sinkhole collector - Go handler for redirecting malicious traffic
// Listens on sinkhole IPs, logs for analysis, doesn't serve real content

func sinkholeHandler(w http.ResponseWriter, r *http.Request) {
    // Extract original intended host from Host header or SNI
    originalHost := r.Host
    clientIP, _, _ := net.SplitHostPort(r.RemoteAddr)
    
    // Fingerprint everything before dropping
    event := AnalysisEvent{
        Timestamp:     time.Now().UTC(),
        SinkholeID:    r.Header.Get("X-Sinkhole-ID"),
        OriginalHost:  originalHost,
        ClientIP:      clientIP,
        UserAgent:     r.UserAgent(),
        JA3:           getJA3(r),
        PayloadHash:   hashBody(r.Body),
        ECS:           r.Header.Get("X-ECS-Subnet"), // passed from DNS layer
    }
    // Async push to Kafka for our threat intel platform
    go kafkaProducer.Publish("sinkhole.events", event)

    // Return minimal 204 to keep bot engaged without reflecting attack
    w.Header().Set("Content-Length", "0")
    w.Header().Set("Cache-Control", "no-store")
    w.WriteHeader(http.StatusNoContent)
}

What We Actually Measure When Millions of Queries Hit the Sinkhole

Redirecting malicious traffic is pointless if you don't learn from it. Our sinkhole isn't a black hole; it's a telescope. Every query and subsequent HTTP/TLS connection that lands on it is logged, sampled, and fed into our detection models. When we first built this, we only counted hits. Now we track the lifecycle of a threat — from the initial DNS query burst, to the HTTP request pattern, to whether the client retries with different query types. That data is what lets our threat mitigation improve the next time that actor tries something new.

During a recent incident involving a phishing kit that was abusing a customer's subdomain, the sinkhole metrics told us more than any firewall log could. We saw that 92% of the clients were reusing the same JA3 fingerprint and that they all tried to POST to /api/login within 200ms of the DNS response. We used that signal to create a WAF rule that blocked the POSTs even before DNS in the next wave. Without the sinkhole data, we would have just seen a DNS spike. With it, we got a full attack playbook.

Below is a snapshot of what we saw before and after sinkholing that specific DGA campaign across our platform. These aren't lab numbers — they're 24-hour aggregates from production, and they show why DNS-level action is so effective for containing malicious traffic before it ever touches your origin.

Deploying Sinkholes in Production: Anycast, RPZ, and Low TTLs

At CDN scale you don't just add a wildcard and hope for the best. Our authoritative tier runs on 280+ anycast PoPs handling peaks north of 12M queries per second, so a bad sinkhole rule can propagate globally in under 3 seconds. We don't touch the base zone files. Instead we layer Response Policy Zones (RPZ) on top of our authoritative backends and feed them from a central control plane. That control plane ingests threat intel feeds, validates them against our own passive DNS, and pushes a signed RPZ feed every 30 seconds to all edge nodes. If you've ever tried to hand-edit BIND zone files across a fleet that size, you know it doesn't scale — you need an automated, versioned feed with instant rollback.

The trickiest part is capacity planning for the sinkhole itself. That sinkhole target — say, sinkhole-collector.cdn.example.net — isn't a black hole, it's a dedicated analysis cluster. We learned the hard way to never point sinkholed names at your production auth IPs. We run sinkhole collectors on separate anycast infrastructure with its own ASN, rate-limited to absorb 500k-800k qps bursts without impacting legitimate DNS. And TTLs matter more than you'd think. We use 60 seconds for all sinkhole records, never 300+ and definitely not 86400. Why? When we had a false positive last March that sinkholed a 2M QPS telemetry domain, a 60s TTL let us unwind it globally in 90 seconds. With a longer TTL, we'd have been serving NXDOMAIN to angry customers for hours. If you're also running signed zones, check out DNSSEC Signing for CDN Infrastructure: Authenticated DNS Without the Performance Hit — we've got to re-sign those synthetic responses or validating resolvers will drop them entirely.

# Production RPZ deployment on PowerDNS Authoritative + BIND secondary
# Central feed master: 10.20.30.53

# PowerDNS authoritative.lua - selective sinkholing with logging
rpzMaster("10.20.30.53", "sinkhole.rpz", {
  refresh = 30,
  maxTTL = 60,
  action = "CNAME sinkhole-collector.cdn.example.net"
})

# BIND 9 response-policy (edge PoPs)
zone "sinkhole.rpz" {
  type secondary;
  masters { 10.20.30.53; };
  file "db.sinkhole.rpz";
  allow-query { none; };
};

options {
  response-policy { zone "sinkhole.rpz"; } qname-wait-recurse no;
  rpz-log yes;
  max-rpz-qname-wait-recurse 32;
};

# db.sinkhole.rpz - distributed feed (TTL deliberately short)
$TTL 60
@ SOA rpz.cdn.example.net. hostmaster.cdn.example.net. 2024051001 30 10 86400 60
  NS localhost.
malware-c2.evil.example.com CNAME sinkhole-collector.cdn.example.net.
*.dga-*.badactor.net CNAME sinkhole-collector.cdn.example.net.
; Wildcard DGA - matches ~15k generated domains/day

Advanced Patterns and Troubleshooting When Things Go Sideways

Once the basics are stable, you'll hit the fun edge cases. Wildcard sinkholing is essential for DGAs that generate 10k-50k domains per day, but you can't just *.evil.com everything — we use regex-based RPZ with Lua policies to match algorithmic patterns while whitelisting known-good subdomains like api.evil.com that turned out to be a compromised but legitimate SaaS. Another pattern we use heavily is geo-sharded sinkholing. Our collectors in North America and APAC sinkhole to different IPs, so we can correlate infection rates by region without hauling petabytes of logs back to one place. We've seen botnets where 73% of callbacks came from three ASNs in Eastern Europe in a 6-hour window — you only catch that if you're sampling at the edge, not aggregating everything centrally.

Troubleshooting is mostly about what you *don't* see. The biggest gotcha is negative caching. If a client got an NXDOMAIN before you sinkholed the domain, they'll cache that negative response for the SOA MINIMUM, often 300-900 seconds, and won't re-query even after your sinkhole is live. You'll think your sinkhole isn't working when it's really just stale cache. Second gotcha: DNSSEC. If you return a synthetic CNAME without a valid RRSIG, any validating resolver (about 38% of our traffic now) will SERVFAIL instead of following your sinkhole. That's why we sign on-the-fly with inline signing. And third, watch your logs. A single sinkholed DGA can generate 40k distinct query names per minute per infected host due to retries. If you're logging every query at full fidelity, you'll drown your Kafka cluster. We sample at 1:100 for high-volume DGAs and log full packets only for low-volume, high-confidence C2s.

And don't forget sinkholing doesn't happen in a vacuum. That same analysis cluster is a juicy target for DNS Amplification Attack Mitigation: Protecting CDN DNS Infrastructure — attackers will try to use your sinkhole as a reflector if you respond with large ANY responses. We enforce response rate limiting (RRL) at 25 responses/sec per /24 and cap responses at 512 bytes for sinkholed names. For resilience, our sinkhole health is tied into the same automation as our origins. See DNS Failover Automation for CDN: Automatic Rerouting When Origins Go Down — if the primary sinkhole collector fails health checks for 10 seconds, traffic automatically reroutes to the secondary analysis site without withdrawing the RPZ rule.

Frequently Asked Questions