DNS Anycast Implementation for CDN: Deploying Resilient Distributed DNS

Back in 2018, we pushed a routine PoP expansion in São Paulo and accidentally withdrew the wrong BGP prefix. Within 90 seconds, DNS query latency for all of South America spiked from 12ms to 280ms as traffic tromboned to Miami. No customer noticed the CDN edge was still up — they just couldn't resolve us. That's when it really clicked for me: you can build the fastest edge network on earth, but if your authoritative DNS isn't anycasted and resilient, you don't have a CDN at all.

Why Unicast DNS Collapses Under Global CDN Load

When I started working on authoritative DNS back in 2012, most mid-size CDNs were still running unicast DNS out of two data centers. It worked until it didn't. A single unicast IP means every resolver on the planet, from Tokyo to Berlin, has to hairpin back to one physical location. You eat the full round-trip penalty, you create a massive DDoS target, and you’re one fiber cut away from a global outage. For a CDN that promises 20ms TTFB, starting with a 150ms DNS lookup is a non-starter.

Anycast flips that model. You announce the same IP prefix — say, 192.0.2.53/24 — from every single PoP simultaneously via BGP. BGP’s natural path selection then pulls each client’s resolver to the topologically nearest announcement. The resolver doesn’t know it’s talking to anycast; it just sends a packet to 192.0.2.53 and the network delivers it to our closest node. When we migrated our primary authoritative tier to anycast in 2015, our global p50 DNS resolution time dropped from 68ms to 14ms overnight without touching a line of DNS software. The network did the work for us.

There's a common misconception that anycast is just about performance. It’s not. The real win is fate-sharing with your edge. If your DNS nodes live in the same facilities as your cache nodes and share the same anycast fabric, a PoP failure becomes invisible. Queries automatically route to the next-nearest PoP. Compare that to unicast with manual failover or low TTLs — you’re always racing DNS caching against detection time. If you're still deciding how to structure domain control, DNS NS Delegation for CDN: Offloading Domain Resolution to Edge Networks is often the precursor decision before you even announce that anycast prefix.

# ExaBGP configuration to announce anycast DNS prefix from each PoP
# /etc/exabgp/exabgp.conf - deployed identically on all authoritative nodes

neighbor 203.0.113.1 {
    router-id 10.10.0.15;
    local-address 10.10.0.15;
    local-as 64512;
    peer-as 64512;

    static {
        # Anycast prefix for ns1.cdn-example.net - 192.0.2.53
        route 192.0.2.0/24 next-hop self community [64512:100];
    }

    process healthcheck {
        run /usr/local/bin/dns-healthcheck.sh;
        encoder json;
    }
}

# Withdraw happens automatically if healthcheck process fails
# No manual intervention - BGP convergence handles rerouting in <30s

BGP Withdrawals and Health Checks That Prevent Blackholing

The biggest mistake I've seen teams make with anycast implementation is announcing the prefix and calling it done. Anycast without surgical health checks is worse than unicast, because BGP will happily keep sending queries to a node where the DNS daemon is hung, the disk is full, or the cache is poisoned. We learned this the hard way during a PowerDNS upgrade that left port 53 open but returning SERVFAIL for 40% of zones. From BGP's perspective, the node was perfectly healthy.

A resilient distributed DNS design needs two layers of checks that can trigger a BGP withdrawal in seconds, not minutes. First, a local daemon check that queries 127.0.0.1 for a known canary zone — something like healthcheck.cdn-example.net — and validates the answer, RCODE, and DNSSEC signature if you’re signing. Second, an external perspective that checks the node as a resolver would, through the anycast IP itself. If either fails three times in a row, you pull the route. Your IGP and upstream transits will reconverge and stop sending you traffic. The node can stay up for debugging, but it’s out of the anycast pool. Don’t try to be clever with graceful de-preferencing via AS-path prepending when the daemon is broken; just withdraw.

You also have to think about state. Authoritative anycast nodes must be stateless and zone-consistent. We rsync signed zonefiles every 30 seconds from a hidden primary and use NOTIFY + IXFR as a fallback, but each edge node must be able to serve the entire zone set independently. There’s no backhaul query on a cache miss like you have with web caches. If you need to serve an apex record without violating the DNS spec, that logic has to be present on every anycast node, which is why many teams combine anycast with DNS CNAME Flattening: Pointing Root Domains to CDN Without Breaking DNS Rules directly on the edge. For more on how HTTP-layer routing interacts with your DNS decisions, the HTTP Working Group Specifications are surprisingly useful context.

#!/bin/bash
# /usr/local/bin/dns-healthcheck.sh - called by ExaBGP healthcheck process
# Exit code 0 = announce, 1 = withdraw

CANARY="healthcheck.cdn-example.net"
EXPECTED="192.0.2.100"
ANYCAST_IP="192.0.2.53"

# Check 1: Local daemon health
RESPONSE=$(dig +short +time=1 +tries=1 @$ANYCAST_IP $CANARY A 2>/dev/null)

if [[ "$RESPONSE" != "$EXPECTED" ]]; then
  echo "withdraw route 192.0.2.0/24 next-hop self" >&2
  logger -t dns-healthcheck "FAIL: Expected $EXPECTED got $RESPONSE - withdrawing"
  exit 1
fi

# Check 2: Validate RCODE and latency
RCODE=$(dig +noall +comments +time=1 @$ANYCAST_IP $CANARY | grep -c "status: NOERROR")
if [[ $RCODE -eq 0 ]]; then
  echo "withdraw route 192.0.2.0/24 next-hop self" >&2
  exit 1
fi

# All good - ensure route is announced
echo "announce route 192.0.2.0/24 next-hop self community [64512:100]"
exit 0

Validating Distributed DNS: What Resilient Actually Means on the Wire

People love to say their DNS is resilient because they have four anycast nodes. That’s not a metric. When we audit a resilient DNS deployment, we're looking at three things you can actually measure: query latency distribution, failure convergence time, and query correctness under load. I don’t care that you have PoPs on six continents if your p95 in Southeast Asia is still 120ms because your BGP community filtering is sending Singapore queries to Los Angeles.

For our last global expansion, we built synthetic monitoring that runs DNS queries from over 400 RIPE Atlas probes every 60 seconds and records not just response time but which PoP answered (via a unique TXT record like id.server.cdn-example.net). That gave us a map of catchment drift — how BGP changes shift traffic between PoPs throughout the day. You’ll be shocked how much catchment moves even when you haven’t touched anything; a Tier 1 peer doing maintenance in Frankfurt can shift 15% of your European queries to Amsterdam in minutes. Your system has to handle that without capacity planning panic. And don’t forget that resolver proximity isn’t user proximity — without EDNS Client Subnet for CDN Geolocation: Routing Users to the Nearest Edge support on your authoritative tier, you might answer quickly from the wrong location.

The performance guides on web.dev Articles hammer this point home: DNS is on the critical path for every single Largest Contentful Paint measurement. Shaving 30ms off DNS p95 has a direct, measurable impact on Core Web Vitals for your customers. So we stopped reporting average DNS latency and started tracking the metrics below as SLOs. If any of them breach, we treat it like a CDN edge outage.

Node-Level Implementation: From BGP Config to Health-Checked Advertisements

Once you've got your global /24 carved out — say, 203.0.113.0/24 for your authoritative DNS — the real work happens inside each PoP. We don't announce the whole /24 from every site. Instead, we loopback-assign a /32 (203.0.113.53/32 is our anycast service address) on every DNS node and let BGP do the distribution. Our standard PoP runs two DNS servers behind ECMP, each running BIRD 2.14 peering with two diverse routers via eBGP. That gives us four paths to withdraw if things go sideways, and it keeps failover under 3 seconds without relying on IGP convergence.

You can't just announce and forget it. Every node runs a local health checker that decides whether that /32 should stay in the table at all. We learned the hard way that a DNS daemon can be "up" on port 53 but serving SERVFAILs because its backend zone transfer broke 10 minutes ago. So our checker doesn't just do a UDP probe — it sends a real A query for a known-good sentinel record (like healthcheck.cdn.example.com) and validates the answer, latency, and RCODE. If it misses three consecutive checks at 2-second intervals, we pull the route. No traffic should ever land on a node that can't answer correctly, and you'll want to pair this with DNSSEC Signing for CDN Infrastructure: Authenticated DNS Without the Performance Hit if you're signing responses at the edge, since a broken signer will also blackhole validation.

# /etc/bird/bird.conf - Anycast DNS node (BIRD 2.x)
router id 192.0.2.10;

protocol direct {
  ipv4;
  interface "lo";
}

# Only announce 203.0.113.53/32 if health check passes
protocol static anycast_dns {
  ipv4;
  route 203.0.113.53/32 via "lo";
  # Controlled by healthcheck script via `birdc`
}

protocol bgp upstream1 {
  local as 64512;
  neighbor 192.0.2.1 as 64500;
  ipv4 {
    import none;
    export where proto = "anycast_dns";
  }
  bfd on; # sub-second failure detection
}

# Health check - run via systemd every 2s
# /usr/local/bin/dns-healthcheck.sh:
# dig @127.0.0.1 healthcheck.cdn.example.com A +short +time=1 | grep -q "192.0.2.53"
# if [ $? -ne 0 ]; then birdc disable anycast_dns; else birdc enable anycast_dns; fi

That birdc disable/enable trick is intentionally crude — it yanks the static route, which triggers an immediate BGP withdrawal. We avoid fancy communities or graceful-shutdown here because when a node is sick, you want it gone *now*. In production across 32 PoPs, this pulls a PoP out of anycast in about 1.8 seconds end-to-end, and we see query failover complete in under 4 seconds for 99th percentile clients. You'll flap if your thresholds are too tight, though — set your failure count to at least 3 and success count to 2 before re-announcing, or a single packet loss will bounce your prefix globally.

Troubleshooting Anycast Pathologies: Flaps, Asymmetry, and Cold PoPs

Anycast looks magic until you have to debug it. The most common pathology we see isn't downtime — it's instability. A node with a flapping link or a CPU-starved BIRD daemon will withdraw and re-announce its /32 every 30-60 seconds, and because BGP dampening is largely disabled for /24s and shorter these days, that flap propagates globally and adds 40-80ms of jitter to your p50 as clients get re-hashed to a PoP 3,000 miles away. We enforce a hard 300-second hold-down on re-announcement after a withdrawal and we rate-limit our health checker so it can't re-enable the protocol more than once per 5 minutes. If a PoP is truly unstable, we'd rather it stay dark and let nearby PoPs absorb 15-20% extra QPS than flap the internet.

The second headache is asymmetry. Remember, the path *to* your anycast node isn't the path *back*. A user in Berlin might hit your Amsterdam PoP on the query, but the response returns via Frankfurt because of hot-potato routing. That breaks any stateful inspection or naive rate-limiting you put in front of DNS. If you're trying to mitigate abuse, don't rely on per-PoP TCP state for DNS — you'll drop legitimate retries. For UDP floods, we've had far more success with response rate limiting (RRL) tuned to 8-12 responses per second per /24 client subnet and aggressive ANY-query filtering. Our writeup on DNS Amplification Attack Mitigation: Protecting CDN DNS Infrastructure covers the RRL tuning that cut our amplification factor from 28x to under 3x without hurting legitimate resolvers.

Finally, watch for cold PoPs. When you bring a new region online — say, Johannesburg — it'll be geographically closest to a huge swath of clients but have a cold cache and no forwarded TCP sessions. For the first 15-20 minutes, you'll see latency *increase* by 10-15ms as resolvers that were previously hitting a warm London node now hit an empty cache in JNB and have to go authoritative. We solve this with pre-warming: we announce with a BGP prepend of 3x for the first 30 minutes and run synthetic queries for the top 10k zones from that PoP to prime the cache. For malicious domains where you *want* that cold behavior, we actually use the same anycast fabric to steer queries to scrubbing nodes — see DNS Sinkholing at CDN Scale: Redirecting Malicious Traffic for Analysis for how we leverage BGP communities to divert sinkholed zones without touching the anycast prefix itself.

Frequently Asked Questions