CDN Failover Testing Playbook: Scripted Scenarios for Disaster Recovery Validation

At 2:17 AM on a Tuesday, our primary origin in us-east-1 went dark for 11 minutes. The CDN was supposed to swing traffic to the standby in eu-west-1 automatically. It didn’t. Half our edge PoPs kept hammering the dead origin until health checks finally timed out, and customers saw 30 seconds of 502s that should have been sub-second. That night taught me that a disaster recovery plan you haven’t scripted and fired for real isn’t a plan — it’s a hope.

Why Most Failover Tests Miss Real-World Failures

I've reviewed dozens of CDN DR validation efforts over the years, and the same pattern shows up again and again. Teams run a clean, manual test during business hours: they shut down an origin server gracefully, wait for the CDN to notice, and declare success when traffic shifts. It looks great in the demo, but it doesn't resemble a real outage. Actual failures are messy. They’re partial packet loss, TLS handshake hangs, slow 5xx trickles, or a BGP withdrawal that only affects certain edge regions. If your CDN Integration Testing Framework Design: Validating Edge Behavior Programmatically only covers the happy-path shutdown, you aren't testing failover at all.

A proper test playbook has to reproduce those dirty failure modes under controlled conditions. We've learned to start every engagement by mapping the actual failure domains: origin health, mid-tier shields, edge PoP isolation, DNS, and certificate validation. Each of those can fail independently, and each triggers a different decision path in the CDN's routing logic. For example, a 503 with a valid `Retry-After` header is often handled differently than a TCP timeout, which is also handled differently than an expired origin certificate. The HTTP Working Group Specifications define how status codes and cache directives should behave, but every CDN provider implements their own interpretation of stale-while-revalidate and stale-if-error, so you can't assume anything until you've measured it.

That means your scripts shouldn't just "turn off the server." They should inject specific, observable faults and then assert on the edge behavior you expect. Here's a simple bash harness we use to automate origin 5xx injection and validate that a secondary origin takes over within our SLO. It’s not fancy, but it catches 80% of misconfigurations before they hit production.

#!/bin/bash
# cdn_failover_smoke.sh - Validate origin failover on 5xx
set -euo pipefail

PRIMARY_ORIGIN="origin-primary.example.com"
FAILOVER_ORIGIN="origin-failover.example.com"
TEST_URL="https://www.example.com/api/health"
MAX_FAILOVER_SECS=5

echo "[*] Injecting 500s on primary origin..."
# Use toxiproxy or iptables to return 500 for 60s
curl -s -X POST http://toxiproxy:8474/proxies/primary/toxics \
  -d '{"name":"500_error","type":"slicer","attributes":{"average_size":0}}' > /dev/null

# Alternative: direct backend fault injection via admin endpoint
# curl -s -X POST "https://${PRIMARY_ORIGIN}/__admin/fault?code=500&duration=60"

START=$(date +%s)
HTTP_CODE=0

while true; do
  HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" -H "Cache-Control: no-cache" $TEST_URL)
  ELAPSED=$(( $(date +%s) - START ))
  
  if [[ "$HTTP_CODE" == "200" ]]; then
    echo "[OK] Failover succeeded in ${ELAPSED}s (served from failover)"
    # Verify via header which origin served request
    curl -s -I $TEST_URL | grep -i "x-origin-served:"
    break
  fi

  if [[ $ELAPSED -gt $MAX_FAILOVER_SECS ]]; then
    echo "[FAIL] Did not fail over within ${MAX_FAILOVER_SECS}s. Last code: $HTTP_CODE"
    exit 1
  fi
  sleep 0.5
done

# Cleanup fault injection
curl -s -X DELETE http://toxiproxy:8474/proxies/primary/toxics/500_error > /dev/null
echo "[*] Test complete. Check edge logs for stale-while-revalidate hits."

Scripted Scenarios That Actually Prove Disaster Recovery

Once you've got the basic harness, you need a library of scripted scenarios. Don't try to build every permutation at once — we've found three to four well-chosen faults cover most disaster recovery risks. I typically start with: hard origin down (TCP refuse), soft origin degradation (5xx at 20% for 2 minutes), and DNS failure for the primary. That last one still catches teams off guard. Your CDN might have the failover origin configured, but if your health checks rely on the same DNS that just broke, failover never triggers.

For each scenario, script both the fault and the assertions. The fault is the easy part; the assertions are where failover testing earns its keep. You’ll want to check status codes from multiple edge locations, time-to-first-successful-failover, cache behavior, and whether client-facing headers like `Age` or `X-Cache` indicate you’re serving stale content as designed. If you're building this into CI, treat it like any other integration test. Our End-to-End CDN Testing: Building Production-Like Test Environments approach spins up a mini-PoP topology with containerized Varnish and Nginx origins, so we can run these scripts on every pull request without touching production.

Here's a Python example we use for DNS-level failover. It blackholes the primary origin's DNS at the edge resolver and validates that requests routed through London, Singapore, and Ashburn all converge on the secondary without serving 5xx to the client. Tools like `dnstox` or just manipulating `/etc/hosts` on the test runner work well in a lab.

import time
import requests
import concurrent.futures
from datetime import datetime

EDGE_COLLECTORS = [
    "https://edge-test-lhr.example.com/check",
    "https://edge-test-sin.example.com/check",
    "https://edge-test-iad.example.com/check"
]
TEST_PATH = "https://www.example.com/static/app.js"
FAILOVER_HEADER = "x-origin-served"
TIMEOUT = 8  # seconds to consider failover successful

def inject_dns_failure():
    # In lab: update CoreDNS to return NXDOMAIN for primary
    print("[*] Injecting DNS NXDOMAIN for origin-primary.example.com")
    requests.post("http://coredns-admin:8080/blackhole",
                  json={"host": "origin-primary.example.com", "action": "nxdomain"})

def check_edge(url):
    try:
        r = requests.get(TEST_PATH, headers={"Host": "www.example.com",
                                             "Cache-Control": "no-cache",
                                             "x-edge-collector": url},
                         timeout=5)
        origin = r.headers.get(FAILOVER_HEADER, "unknown")
        return url, r.status_code, origin, r.elapsed.total_seconds()
    except Exception as e:
        return url, 0, str(e), 999

def run_test():
    inject_dns_failure()
    start = datetime.now()
    print(f"[*] Fault injected at {start.isoformat()}, polling edges...")

    with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor:
        for i in range(TIMEOUT * 2):
            futures = [executor.submit(check_edge, u) for u in EDGE_COLLECTORS]
            results = [f.result() for f in futures]
            
            all_ok = all(code == 200 and "failover" in origin for _, code, origin, _ in results)
            if all_ok:
                elapsed = (datetime.now() - start).total_seconds()
                print(f"[OK] DNS failover validated in {elapsed:.2f}s")
                for u, code, origin, lat in results:
                    print(f"  {u} -> {code} from {origin} in {lat:.3f}s")
                return True
            time.sleep(0.5)

    print("[FAIL] Not all edges failed over within timeout")
    return False

if __name__ == "__main__":
    success = run_test()
    # Always clean up DNS injection
    requests.post("http://coredns-admin:8080/blackhole",
                  json={"host": "origin-primary.example.com", "action": "clear"})
    exit(0 if success else 1)

Defining Pass/Fail: Metrics That Make CDN DR Validation Credible

You can’t call a failover test green just because you eventually got a 200. Without hard thresholds, every test becomes a judgment call. We define our pass/fail criteria before we run a single script, and we tie them directly to what users actually experience. That means looking beyond origin availability and focusing on edge-perceived latency, error volume during the transition window, and cache continuity. Guidance from web.dev Articles on Core Web Vitals is useful here — a failover that adds 2 seconds of Time to First Byte is technically successful but still damages your UX.

In our playbook, we log three telemetry streams for every run: synthetic checks from external catchpoints (like ThousandEyes or Catchpoint), CDN access logs with `origin_status` and `cache_status`, and Real User Monitoring for the test domain. The synthetics give us region-by-region failover time, the logs tell us whether we served stale or went to origin, and RUM confirms there wasn't a hidden spike for actual browsers. When we started correlating these after a failed drill two years ago, we found our US-West PoPs were 4x slower to fail over because health check intervals were tuned for capacity, not resiliency.

For teams building their own CDN Performance Testing Methodology: Load Generation, Measurement, and Analysis, these thresholds give you a solid baseline. Adjust them for your SLOs, but don't ship a DR validation that doesn't measure them. If you can't graph the failover window, you can't prove you met it.

#!/bin/bash
set -euo pipefail

PRIMARY_ORIGIN="origin-primary.example.com"
FAILOVER_ORIGIN="origin-failover.example.com"
EDGE_HOST="www.example.com"
MAX_FAILOVER_SECS=45
TEST_URL="https://${EDGE_HOST}/static/app.js"

echo "[1/4] Establishing baseline - warming cache & checking health"
curl -s -o /dev/null -w "%{http_code}" $TEST_URL | grep -q 200
for POP in "us-east" "eu-west" "ap-southeast"; do
  curl -s -H "x-debug-pop: $POP" $TEST_URL --resolve "$EDGE_HOST:443:104.16.0.1" | grep -q "x-cache: HIT"
done

echo "[2/4] Injecting fault - blocking primary origin at edge ACL"
# Use your CDN's API - Cloudflare, Fastly, Akamai all support this
cdn_api block-origin --origin $PRIMARY_ORIGIN --ttl 600 &
BLOCK_PID=$!

echo "[3/4] Validating failover within ${MAX_FAILOVER_SECS}s"
START=$(date +%s)
until curl -sf -m 5 $TEST_URL -H "x-failover-test: 1" > /dev/null; do
  ELAPSED=$(( $(date +%s) - START ))
  if [ $ELAPSED -gt $MAX_FAILOVER_SECS ]; then
    echo "FAIL: Failover exceeded ${MAX_FAILOVER_SECS}s - check origin health checks (10s interval / 3 fails?)"
    kill $BLOCK_PID; exit 1
  fi
  sleep 2
done

echo "PASS: Failover in $(( $(date +%s) - START ))s - verifying cache behavior"
# Ensure we're serving stale/failover and not bypassing cache
curl -s -D - $TEST_URL -o /dev/null | grep -qi "x-cache:.*HIT\|x-serve-stale: 1"
curl -s -D - $TEST_URL | grep -q "x-origin: $FAILOVER_ORIGIN"

echo "[4/4] Rollback & post-check"
kill $BLOCK_PID
sleep 10
dig +short $EDGE_HOST | grep -q "104.16"
echo "Done - check your RUM for 5xx spike < 0.1% in next 5m"