Back in 2018, I watched a major e-commerce site stay dark for 42 minutes even though they had three healthy origins sitting idle. The problem wasn't their servers — it was DNS. Their authoritative DNS kept handing out the IP of a failed PoP in Ashburn because nobody had told it the endpoint was dead. That's when I learned the hard way: if your DNS doesn't probe before it routes, you're just gambling with TTLs.
Why Blind DNS Routing Fails at CDN Scale
Traditional DNS is dumb by design. It answers the question "where is this hostname?" but it has no opinion about whether that destination is actually serving traffic. For a simple website with one origin, that's fine. For a CDN with dozens of edge clusters and multiple origins behind each one, it's a disaster waiting to happen. I've seen teams spend weeks tuning Anycast and EDNS Client Subnet for CDN Geolocation: Routing Users to the Nearest Edge only to have a single failed health check take out an entire region because their DNS had no fallback logic.
We dealt with this exact scenario when we were rolling out a new edge in Frankfurt. The servers were passing local checks, but an upstream transit provider was blackholing traffic to our /32. Our monitoring inside the PoP showed all green, yet users in eastern Europe were hitting timeouts. The authoritative DNS kept returning that PoP's IP because, from its perspective, nothing was wrong. That's the gap health checking fills — it gives your DNS a live view of the world, not just a static zone file. Without it, you're relying on clients to retry or on short TTLs to save you, and neither is reliable when you're pushing 2M+ queries per second.
# Example health check definition for an authoritative DNS control plane
# This isn't BIND syntax — it's what most CDN control planes compile down to
health_check "edge-fra-03" {
endpoint = "203.0.113.45"
protocol = "https"
port = 443
path = "/cdn-health"
# Don't just check TCP — verify the application
expected_status = 200
expected_body_contains = "ok"
host_header = "health.example.net"
interval = 5s
timeout = 2s
failure_threshold = 3 # Mark down after 3 consecutive failures
success_threshold = 2 # Require 2 successes to mark up again
# Where to probe FROM — critical for CDN
probe_sources = ["ams", "fra", "waw", "vie"]
quorum = 2 # At least 2 vantage points must agree
}Building Probes That Tell the Truth
A lot of teams start with a simple TCP connect or ICMP ping and call it a day. I've made that mistake too. Early in my career we used TCP 443 checks for an edge that terminated TLS, and we kept marking nodes as healthy even though the Nginx worker pool was exhausted and returning 503s to every real request. Your probe must mimic what a real user does, otherwise you're measuring the wrong thing. For HTTP-based CDN endpoints, that means a full TLS handshake, sending a Host header, and validating both status code and body. The HTTP Working Group Specifications are useful here for defining what a correct response should actually look like, especially around cache-related headers that can mask origin failures.
The other trap is probing from one location. If your prober lives in us-east-1 and your endpoint is in Singapore, a single network blip looks like an endpoint failure. We run distributed probers for every endpoint and require a quorum before we pull an IP from DNS. In practice, that cut our false positives by about 80% when we switched from single-site to four-site quorum. You also need to think about what you're not checking — we once forgot to disable follow-redirects and our probes were happily following a 302 to a different PoP and reporting success while the original node was down. Now we explicitly fail on any 3xx that points off-host. For background on how this fits into broader delegation strategies, see DNS NS Delegation for CDN: Offloading Domain Resolution to Edge Networks, and the guidance on web.dev Articles helped us shape realistic synthetic traffic patterns.
#!/usr/bin/env python3
# Simplified distributed health prober — the logic our edge controller runs
import http.client
import ssl
import time
def probe_endpoint(ip, host, path="/cdn-health", timeout=2.0):
ctx = ssl.create_default_context()
conn = http.client.HTTPSConnection(ip, timeout=timeout, context=ctx)
try:
# Use IP as connection target, but send real Host header
# This verifies SNI + Host routing + cert validity
conn.request("GET", path, headers={"Host": host, "User-Agent": "cdn-prober/1.0"})
resp = conn.getresponse()
body = resp.read(256).decode(errors="ignore")
# Strict validation — don't accept redirects or generic 200s
if resp.status != 200:
return False, f"bad status {resp.status}"
if "ok" not in body.lower():
return False, f"body mismatch: {body[:50]}"
if resp.getheader("x-cache-status") == "ERROR":
return False, "cache layer reported error"
return True, "healthy"
except Exception as e:
return False, str(e)
finally:
conn.close()
# Called from multiple regions; aggregator requires quorum
# result = all(probe_endpoint(...) for prober in quorum_members)Dialing In Intervals, Timeouts, and Threshold Logic
There’s no perfect set of numbers, but there are wrong ones. If your interval is 30 seconds and your failure threshold is 5, you’ve just baked in a 2.5-minute outage before DNS reacts. Users won’t wait that long. On the other hand, if you probe every second with a 500ms timeout and a threshold of 1, you’ll flap constantly and your DNS responses will churn, blowing up your cache hit ratio and confusing resolvers. We’ve settled on a 5-second interval with a 2-second timeout for most edge HTTP checks, which gives us detection in about 15 seconds without getting twitchy over a single dropped packet.
Thresholds are where you make or break stability. We use 3 consecutive failures to mark an endpoint down and 2 consecutive successes to bring it back. That hysteresis prevents flapping when an endpoint is on the edge of capacity. And you’ve got to coordinate this with your TTL strategy. A 30-second DNS TTL doesn’t help if your health check takes 60 seconds to fail over. We tie our health check interval to our TTL — our TTL is always at least 2x the detection time. If you’re using DNS CNAME Flattening: Pointing Root Domains to CDN Without Breaking DNS Rules at the apex, remember that flattened records often get cached longer at recursive resolvers, so your effective failover time may be longer than your authoritative TTL suggests.
Below is the baseline we use for most production CDN endpoints. We’ll tune these per PoP based on latency and error budgets, but this table is our starting point for any new region launch.
| Parameter | Default Value | Impact on Failover | When to Adjust |
|---|---|---|---|
| Probe Interval | 5 seconds | Detection time = interval × failure_threshold | Lower to 2-3s for premium, low-latency tiers |
| Probe Timeout | 2 seconds | Slow endpoints marked down faster | Raise to 4s for high-latency regions (APAC, SA) |
| Failure Threshold | 3 consecutive failures | ~15 sec total to withdraw IP from DNS | Raise to 5 if you see flap due to packet loss |
| Success Threshold | 2 consecutive successes | ~10 sec to reintroduce IP | Raise to 3 for endpoints recovering from overload |
| DNS TTL (healthy) | 30 seconds | How long bad IPs stay cached at resolvers | Drop to 10s during incidents, raise to 60s when stable |
Wiring Health Checks Into Your DNS Control Plane
Getting probes right in the lab is one thing. Making them drive actual DNS answers without flapping your entire edge is another. Most teams fail here because they wire health checks directly to record removal. Don't. You want a control plane that decouples probing from publication and gives you a damping layer in between. We run a dedicated health aggregator that collects probe results from 5-7 PoPs per region, applies threshold logic centrally, and only then pushes a state change to our authoritative DNS via API.
That indirection lets you enforce quorum and avoid the classic split-brain where a single prober with a bad NIC yanks a healthy origin. Our rule is simple: an endpoint is marked down only if 3+ geographically diverse probers agree it's failed for two consecutive intervals. With a 10-second interval and a 2-second timeout, that means 20-30 seconds to de-list, which is fast enough to dodge a real outage but slow enough to ride out a transient blip. And keep your TTLs tight — we publish CDN endpoint A-records with a 30-second TTL, 15 seconds during an active failover event. Anything longer and you're just waiting for cached failures to expire. If you're not thinking about authenticated responses while you're retooling your DNS pipeline, it's worth reviewing DNSSEC Signing for CDN Infrastructure: Authenticated DNS Without the Performance Hit before you automate record churn.
The glue code itself isn't magic, but it has to be explicit about state transitions. Here's a stripped-down version of our aggregator's decision loop in Python — this is what actually decides whether to call the DNS API:
# Aggregator decision logic - runs every probe interval (10s)
ENDPOINTS = ["203.0.113.10", "203.0.113.11", "198.51.100.42"]
FAIL_THRESHOLD = 3 # consecutive failures to mark down
SUCCESS_THRESHOLD = 2 # consecutive successes to mark back up
QUORUM = 3 # probers that must agree
def evaluate_endpoint(endpoint, prober_results):
# prober_results: dict[prober_id -> bool (True=healthy)]
healthy_votes = sum(1 for v in prober_results.values() if v)
failed_votes = len(prober_results) - healthy_votes
# Quorum-based state machine with hysteresis
if state[endpoint] == "healthy":
if failed_votes >= QUORUM:
failures[endpoint] += 1
if failures[endpoint] >= FAIL_THRESHOLD:
state[endpoint] = "draining"
# Stop routing NEW connections, keep TTL at 15s
dns_api.update_record(endpoint, weight=0, ttl=15)
time.sleep(30) # drain period
dns_api.remove_record(endpoint)
state[endpoint] = "down"
else:
failures[endpoint] = 0
else: # currently down
if healthy_votes >= QUORUM:
successes[endpoint] += 1
if successes[endpoint] >= SUCCESS_THRESHOLD:
dns_api.add_record(endpoint, weight=100, ttl=30)
state[endpoint] = "healthy"
Advanced Patterns and Debugging When Probes Lie
Once the basics are stable, you'll hit the subtle failures. The most common is the “healthy but useless” endpoint — it passes TCP and HTTP 200 checks, yet p95 latency to real users is 850ms because its upstream cache is thrashing. We solved this with layered health: a light L4 check every 10 seconds for liveness, plus a heavier L7 check every 30 seconds that fetches a 16KB canary object and validates TTFB < 250ms and full download < 600ms. If either layer fails twice in a row, we shed 50% of traffic via weighted routing before fully de-listing. That gradual shed has cut our false-positive removals by about 60%.
Another pattern worth stealing is shadow probing for canary PoPs. When you bring a new CDN node online, don't add it to the pool immediately. Let your probers hit it for 5-10 minutes in shadow mode — collecting success rate, latency, and TLS handshake time — without serving production traffic. We've caught cold-cache origins and misconfigured WAF rules this way that would've spiked 5xx rates by 2-3% if we'd just thrown them into the rotation. On the security side, your health checks can also surface abuse patterns early. If you see probes triggering rate-limits or getting reflected responses, cross-reference with your DNS Amplification Attack Mitigation: Protecting CDN DNS Infrastructure playbook before you assume it's an endpoint failure.
When troubleshooting flaps, start with correlation, not logs. If all probers in eu-west fail but us-east probers stay green, it's rarely the origin — it's a transit issue or a targeted block. Check for anycast routing shifts and then look at whether malicious domains are polluting your view. We once chased a “flaky PoP” for two days that turned out to be DNS sinkholing responses contaminating our resolver cache, which our guide on DNS Sinkholing at CDN Scale: Redirecting Malicious Traffic for Analysis now covers as a pre-check. Always export probe metrics — success ratio, timeout rate, and latency histograms — to your observability stack. If you're seeing >0.5% timeouts with <1% HTTP 5xx, lower your timeout from 2s to 1.5s; you're likely waiting too long and delaying failover.
Frequently Asked Questions
How many consecutive failures should trigger DNS failover?
For most CDN workloads, 3 consecutive failures from a quorum of probers is the sweet spot. With a 10-second interval, that gives you a 30-second detection window — fast enough to beat user impact, but dampened enough to ignore single packet loss events. Don't use a single failure; at scale, you'll yank healthy nodes 10-15 times a day on noise alone. Require 2 consecutive successes before re-adding.
Should health checks use the same anycast IP that users do?
No — and this is a common mistake. Probe the unicast origin IP or the node-specific hostname directly. If you probe the anycast address, your check will just be routed to the nearest healthy PoP and you'll never detect a failure in the farther one. Use unicast for truth, use anycast for user traffic. Keep a mapping table in your aggregator so DNS records still publish the anycast VIPs.
What's the right TTL for health-checked CDN records?
30 seconds is a solid default, dropping to 10-15 seconds during failover. Lower than 10 seconds and you'll hammer your authoritative DNS — we saw query volume jump 4x when we tried 5 seconds globally. Higher than 60 seconds and your failover is too slow; users will cache the bad IP for a full minute. Use short TTLs only on the A/AAAA records that are health-checked, not your entire zone.
Why are my probes healthy but users still see 503s?
Your probe is probably too shallow. A TCP connect to port 443 succeeding doesn't mean the cache, app, or upstream origin is healthy. Add an L7 check that requests an actual object through the full CDN stack — including cache lookup and origin fetch — and validate both status code and body hash. Also check that your probe's Host header and SNI match real user traffic; we've seen origins return 200 to a generic probe but 503 to the real vhost.