DNS Load Testing for CDN Infrastructure: Stress-Testing Your Resolution Layer

Back in 2019, I watched a perfectly healthy CDN edge cluster sit idle while users saw timeouts. The problem wasn’t bandwidth or cache hit ratio — our authoritative DNS couldn’t keep up with a 40x spike in queries after a major customer pushed a config change that busted TTLs. That incident taught me a lesson I’ve carried since: you can have the best edge network in the world, but if your resolution layer falls over, nothing else matters. DNS load testing isn’t optional for CDN infrastructure; it’s how you prove your control plane can survive what the real internet throws at it.

Why the Resolution Layer Breaks Long Before Your Edge Does

Most teams stress-test their HTTP stack relentlessly. They’ll simulate thousands of concurrent connections, saturate origin links, and measure Time to First Byte until they’re blue in the face. But DNS gets treated like plumbing — assumed to just work. I’ve seen that assumption fail more than once, and it always fails under the same conditions: low TTLs, high query fan-out, and unexpected client retry behavior.

Here’s what makes DNS NS Delegation for CDN: Offloading Domain Resolution to Edge Networks so tricky to get right. When you delegate a customer’s domain to your anycast DNS footprint, you’re not just answering one query per page load. A single user session might trigger lookups for the apex, www, static assets, API endpoints, and then retries from resolvers that didn’t get an answer fast enough. Multiply that by millions of users behind recursive resolvers like Google Public DNS or Cloudflare’s 1.1.1.1, and your authoritative servers see a very different pattern than what you see in your CDN access logs. Add in EDNS Client Subnet for CDN Geolocation: Routing Users to the Nearest Edge — which increases response variance and cacheability — and your per-query compute cost goes up just when query volume peaks.

That’s why resolution performance has to be tested independently from your delivery layer. I learned to run DNS capacity tests on isolated infrastructure first, without any CDN caching logic in the path. You want to know your raw QPS ceiling, your latency distribution under load, and where packet loss starts. If you don’t know those numbers, you’re guessing at capacity, and guessing is a bad strategy when a flash sale or a DDoS forces every client to re-resolve at once.

# Baseline authoritative QPS test with dnsperf
# queryfile.txt contains: example.cdn-test.net A, example.cdn-test.net AAAA
dnsperf -s 203.0.113.10 -p 53 -d queryfile.txt \
  -c 100 -T 30 -Q 75000 \
  -l 60 -S 1

# Key flags:
# -c 100 = 100 concurrent clients
# -Q 75000 = target 75k QPS
# -S 1 = print stats every second to catch latency spikes

Designing a DNS Stress Test That Actually Looks Like Production

A vanilla DNS stress test that just floods your server with the same A record query isn’t going to tell you much. Real traffic is messy. I’ve found that the most useful tests mix query types, randomize qnames, and simulate resolver behavior — including the bad behavior. Your test harness needs to send a blend of A, AAAA, CNAME, and even TXT queries, with a realistic ratio of cache hits to cache misses on your authoritative side. If you only test hot-cache answers, you’ll miss the database or control-plane lookup cost that kills you during a cold-start event.

When we re-architected our resolution layer two years ago, we made a point of replaying anonymized production query logs instead of synthetic traffic. We took a 10-minute window from a peak event, sanitized it, and used it as the template for our load generation. The difference was eye-opening. Synthetic tests showed we could handle 500k QPS per PoP. Replay tests showed we started dropping at 280k QPS because of the long tail of non-existent domains and DNS CNAME Flattening: Pointing Root Domains to CDN Without Breaking DNS Rules logic that required additional lookups. That flattening path is expensive — it trades RFC compliance for user convenience — and it only shows up if your test data includes apex queries.

Don’t forget to model UDP versus TCP and truncated responses. Modern DNS load testing has to account for larger responses from DNSSEC and ECS that force TCP fallback. I typically run two parallel generators: one for high-volume UDP and a smaller, persistent TCP stream to measure handshake and connection reuse overhead. The HTTP Working Group Specifications (https://httpwg.org/specs/) and resources on web.dev Articles (https://web.dev/articles/) were helpful references when we were correlating DNS latency with actual page load impact — slow resolution directly inflates connection setup time before any HTTP request even happens.

# Realistic mixed-query test with dnsperf + Flamethrower
# Generate 1M unique queries to bust cache and test DB path

# 1. Build query file with qname randomization
awk '{print rand() "-asset-" $0}' prod-queries-sample.txt > mixed-queries.txt

# 2. Run Flamethrower for high-concurrency UDP + TCP fallback check
flamethrower 203.0.113.10 -p 53 -c 200 -r 100000 \
  -q mixed-queries.txt -P udp -l 120 \
  --qps 150000 --timeout 2.0 --delay 0

# 3. In parallel, run TCP-only stream to measure fallback cost
dnsperf -s 203.0.113.10 -p 53 -d mixed-queries.txt \
  -m tcp -c 50 -Q 5000 -l 120

What Healthy Resolution Performance Looks Like Under Load

Once you’ve got the traffic model right, you need clear thresholds. I don’t care much about average latency — averages hide the pain that real users feel. What I watch is p95 and p99 latency, query timeout rate, and the point where your server starts sending truncated or SERVFAIL responses. If your p99 goes from 12ms to 80ms at 60% of your target QPS, you’ve got a CPU or lock contention problem you need to fix before you add more traffic. I’ve seen teams celebrate hitting 1M QPS while ignoring that 2% of queries were timing out; that 2% translates to millions of failed page loads at CDN scale.

We baseline every PoP before it goes live and re-test quarterly. Your DNS capacity shouldn’t be a single global number — it varies by PoP hardware, upstream database latency, and even kernel tuning. I keep a simple dashboard that tracks queries per second per core, responses per second, and drops at the NIC. If your resolution performance degrades linearly with QPS, you can scale horizontally. If it falls off a cliff, you’ve got a bottleneck in your software or your data path. That cliff is what a proper DNS stress test is meant to find, and you want to find it in a lab, not during a Super Bowl ad.

Below is the pass/fail criteria we’ve used for the last three years for our edge authoritative clusters. These aren’t universal — your numbers will depend on hardware and anycast setup — but they’re a solid starting point for an intermediate team trying to define Service Level Objectives for their DNS tier. Anything worse than the “Review” column triggers an immediate capacity review, not after the next incident.

Building a Production-Safe Load Rig That Doesn't Take Down Production

You've got the test plan on paper — now you need to run it without becoming the outage you're trying to prevent. The biggest mistake we see teams make is pointing their load generator at the production authoritative VIPs and letting it rip. Don't. Even if you've got rate-limiting in place, you'll pollute caches, skew your real query analytics, and if your generator misbehaves, you'll trigger the exact DNS Amplification Attack Mitigation: Protecting CDN DNS Infrastructure throttles you were trying to measure.

Instead, stand up a shadow authoritative stack. Clone your production zone files, configs, and anycast routing policy onto isolated hardware or a separate container fleet that mirrors production's CPU, memory, and network caps. Your load generators should live in at least three geographically dispersed regions — we typically use one in us-east, one in eu-west, and one in ap-southeast — to catch anycast routing weirdness and EDNS Client Subnet handling. Start low and ramp in steps: 5k QPS for 10 minutes to warm caches and establish a baseline, then step to 25k, 50k, and 100k QPS, holding each level for 15 minutes. We've found that 70-80% of latent bugs only show up after that 12-15 minute mark when slab allocators and conntrack tables start to fill.

Your query mix is everything. Don't just replay pcap. Generate a synthetic query file that reflects your real distribution: 85% valid A/AAAA hits, 10% NXDOMAIN for random subdomains (to hammer your negative cache), 3% TXT/HTTPS queries, and 2% deliberately malformed or DNSSEC-flagged queries if you're validating. That last 2% is critical if you're running DNSSEC Signing for CDN Infrastructure: Authenticated DNS Without the Performance Hit — signature validation adds 15-30% CPU overhead under load, and you won't see it with plain A queries. Here's a minimal harness we use with dnsperf and flamethrower to drive that mix without writing a custom generator:

# 1. Build your queryfile (queryfile.txt) with realistic distribution
# Format: <domain> <type>
cdn.example.com A
cdn.example.com AAAA
static-93742.cdn.example.com A
nx-8f3a9b2c.cdn.example.com A
cdn.example.com TXT
cdn.example.com RRSIG

# 2. Run a stepped load test with dnsperf (accurate latency histograms)
dnsperf -s 203.0.113.10 -p 53 -d queryfile.txt \
  -c 100 -Q 50000 -l 900 -e

# 3. Or for high-QPS + TCP fallback testing with flamethrower
flamethrower 203.0.113.10:53 -q queryfile.txt \
  --qps 75000 --concurrency 200 --timeout 2000 \
  --rate 0 --edns --dnssec --tcp-fallback \
  --latency-histogram --stats-interval 5

# 4. Monitor from the server side during the run
# watch -n 1 'rndc stats && grep -E "queries|dropped|truncated" /var/log/named/stats'

When Your Lab Numbers Lie: Cache Traps, TCP Meltdowns, and Phantom Drops

So you hit 100k QPS in the lab with a beautiful 8ms p95 and you ship it. Then the first real traffic spike hits and latency triples. What happened? Nine times out of ten, it's a cache trap. Your lab test was 90% cache hits because you were querying the same 10k hostnames over and over. Real CDN traffic has a long tail — we've measured that even with a 60-second TTL, a realistic 50k QPS stream will have a 35-45% cache miss rate at the recursive layer if you're not careful with query randomization. If your auth servers aren't sized for that miss rate, you'll see CPU soft-lock at 85-90% and start dropping UDP silently. The fix is to salt your queryfile with at least 100k unique labels per minute and monitor your auth's cache hit ratio separately from your p95.

The second phantom bottleneck is TCP. Everyone tests UDP and assumes they're done. At around 3-5% of your total queries, you'll get truncation (TC=1) and a TCP retry — that number jumps to 12-15% if you're serving large DNSSEC or HTTPS/SVCB records. A single TCP handshake is 3x the memory and 10x the CPU of a UDP query, and most Linux defaults (like net.ipv4.tcp_max_syn_backlog = 128 and somaxconn) will choke long before your DNS daemon does. We tune our auth fleet to 4096 backlog and run with net.core.rmem_max at 16MB specifically for this. If you don't test TCP fallback at scale, you won't see the SYN queue drops until customers do.

Finally, don't ignore what your own security policies do under load. If you've implemented DNS Sinkholing at CDN Scale: Redirecting Malicious Traffic for Analysis, your sinkhole logic (usually a response policy zone or Lua hook) runs inline on every single query. Under 60k+ QPS, a poorly indexed RPZ with 200k+ entries can add 18-25ms per lookup. We isolate that by load testing with the RPZ enabled *and* disabled and comparing the delta. If the delta is more than 5ms at p99, it's time to shard or pre-compile that zone. Your load test should prove your mitigation doesn't become the bottleneck.

Frequently Asked Questions