Chaos Engineering for CDN: Building Resilient Edge Infrastructure

Your CDN handles millions of requests without incident — until it doesn't. The origin goes down, a cache purge stampedes the backend, or a misconfigured edge function starts returning 500 errors. You've designed for these scenarios, but have you actually tested them? Chaos engineering closes the gap between "we think this failover works" and "we've proven it works under real conditions." Applied to CDN infrastructure, chaos engineering validates that your edge caching, origin failover, load balancing, and error handling actually behave correctly when components fail.

Why CDN Infrastructure Needs Chaos Testing

CDN architectures create an illusion of simplicity. Your application makes a request, the CDN routes it to the nearest edge PoP, and the response comes back fast. Behind that simplicity is a complex system of DNS resolution, TLS termination, cache lookup, origin selection, request routing, response transformation, and cache storage. Each step can fail independently, and failures compound in ways that are impossible to predict without testing.

Consider a common assumption: "If the origin goes down, the CDN will serve cached content." This is true — for content that's already cached with unexpired TTLs. But what about cache misses? Dynamic API responses? Content with short TTLs that expired minutes before the outage? What about stale-while-revalidate — does your CDN actually honor it, and for how long? Chaos engineering answers these questions with evidence instead of assumptions.

CDN failures are also distinctly hard to reproduce in staging environments. You can't replicate the cache state of 200+ edge PoPs, the geographic distribution of real users, or the traffic patterns that make certain failure modes dangerous. This is why controlled chaos experiments in production — with proper safeguards — provide information that no other testing method can.

Experiment 1: Origin Outage Resilience

The most fundamental chaos experiment for CDN infrastructure is origin failure. Kill the origin and observe what happens at the edge. This tests your entire resilience stack: cache TTLs, stale-while-revalidate behavior, custom error pages, origin failover, and health check response times.

# chaos-origin-outage.py
import requests
import time
import subprocess
from datetime import datetime

CDN_URL = "https://www.example.com"
ORIGIN_HOST = "origin-1.example.com"
TEST_PATHS = ["/", "/api/products", "/images/hero.webp", "/blog/latest"]
DURATION_SECONDS = 300  # 5 minute outage simulation
ABORT_ERROR_RATE = 0.50  # Abort if >50% errors

def baseline_measurements():
    """Measure normal behavior before experiment"""
    results = []
    for path in TEST_PATHS:
        resp = requests.get(f"{CDN_URL}{path}", timeout=10)
        results.append({
            "path": path,
            "status": resp.status_code,
            "latency_ms": resp.elapsed.total_seconds() * 1000,
            "cache_status": resp.headers.get("CF-Cache-Status", "unknown"),
            "age": resp.headers.get("Age", "0")
        })
    return results

def kill_origin():
    """Simulate origin outage via iptables"""
    subprocess.run([
        "ssh", ORIGIN_HOST,
        "sudo iptables -A INPUT -p tcp --dport 443 -j DROP"
    ])

def restore_origin():
    """Restore origin connectivity"""
    subprocess.run([
        "ssh", ORIGIN_HOST,
        "sudo iptables -D INPUT -p tcp --dport 443 -j DROP"
    ])

def run_experiment():
    print("=== BASELINE ===")
    baseline = baseline_measurements()
    for r in baseline:
        print(f"  {r['path']}: {r['status']} ({r['cache_status']}) "
              f"{r['latency_ms']:.0f}ms")

    print(f"\n=== KILLING ORIGIN at {datetime.now()} ===")
    kill_origin()

    errors = 0
    total = 0
    try:
        for tick in range(DURATION_SECONDS // 5):
            time.sleep(5)
            for path in TEST_PATHS:
                total += 1
                try:
                    resp = requests.get(f"{CDN_URL}{path}", timeout=10)
                    cache = resp.headers.get("CF-Cache-Status", "?")
                    if resp.status_code >= 500:
                        errors += 1
                    print(f"  t={tick*5:3d}s {path}: "
                          f"{resp.status_code} ({cache})")
                except requests.Timeout:
                    errors += 1
                    print(f"  t={tick*5:3d}s {path}: TIMEOUT")

            error_rate = errors / total
            if error_rate > ABORT_ERROR_RATE:
                print(f"\n!!! ABORT: error rate {error_rate:.1%} "
                      f"exceeds threshold !!!")
                break

    finally:
        print(f"\n=== RESTORING ORIGIN at {datetime.now()} ===")
        restore_origin()

    print(f"\n=== RESULTS ===")
    print(f"Total requests: {total}")
    print(f"Errors: {errors} ({errors/total:.1%})")

run_experiment()

What to Look For

• Cached responses should continue serving normally with HIT cache status. If cached URLs start returning errors, your CDN's stale-serving configuration is broken.
• Cache misses will fail. The question is how they fail — does the CDN return a branded error page, a generic 502, or does it hang until timeout? Configure origin shield and custom error pages to handle this gracefully.
• TTL expiration during outage is the critical moment. When a cached response's TTL expires while the origin is down, does the CDN serve stale content (if stale-if-error is configured) or immediately return an error?
• Recovery time after origin restoration — how long until all edge PoPs detect the origin is healthy again and resume normal operation?

Experiment 2: Cache Purge Storm

A mass cache purge — intentional or accidental — forces every request back to the origin. This is one of the most dangerous CDN failure scenarios because it turns your CDN's distributed caching layer into a distributed DDoS against your own origin.

# chaos-cache-purge-storm.py
import requests
import threading
import time

CDN_API = "https://api.cdn-provider.com/v1"
ZONE_ID = "your-zone-id"
API_TOKEN = "your-api-token"

def purge_everything():
    """Trigger a complete cache purge"""
    resp = requests.post(
        f"{CDN_API}/zones/{ZONE_ID}/purge_cache",
        json={"purge_everything": True},
        headers={"Authorization": f"Bearer {API_TOKEN}"}
    )
    return resp.json()

def measure_origin_load():
    """Monitor origin request rate during purge storm"""
    while monitoring:
        metrics = get_origin_metrics()
        print(f"  Origin RPS: {metrics['rps']:.0f} | "
              f"CPU: {metrics['cpu']:.0f}% | "
              f"Queue: {metrics['queue_depth']}")
        time.sleep(2)

# Warm the cache first
print("Warming cache with top 1000 URLs...")
warm_cache(top_urls[:1000])

# Start monitoring
monitoring = True
monitor_thread = threading.Thread(target=measure_origin_load)
monitor_thread.start()

# Purge and observe
print("Purging entire cache...")
purge_everything()

# Watch origin for 5 minutes
time.sleep(300)
monitoring = False
monitor_thread.join()

This experiment reveals whether your origin can handle the thundering herd after a full purge. Key mitigations include request collapsing (coalescing duplicate cache-miss requests into a single origin fetch), origin rate limiting, and circuit breakers that return stale or synthetic responses when the origin is overwhelmed.

Experiment 3: Edge Function Failure

Edge functions add powerful capabilities — and powerful failure modes. A runtime exception in a Cloudflare Worker or Lambda@Edge function can break request processing for every request that matches the function's route pattern.

// chaos-edge-function.js
// Deploy this to a test route to simulate edge function failures
export default {
  async fetch(request, env) {
    const chaosMode = request.headers.get("X-Chaos-Mode");

    switch (chaosMode) {
      case "exception":
        throw new Error("Simulated runtime exception");

      case "timeout":
        // Simulate a function that never returns
        await new Promise(resolve => setTimeout(resolve, 30000));
        break;

      case "memory":
        // Simulate memory pressure
        const leak = [];
        while (true) {
          leak.push(new ArrayBuffer(1024 * 1024));
        }

      case "slow":
        // Add 2 seconds of artificial latency
        await new Promise(r => setTimeout(r, 2000));
        return fetch(request);

      case "wrong-origin":
        // Route to wrong origin (misconfiguration simulation)
        const url = new URL(request.url);
        url.hostname = "nonexistent-origin.example.com";
        return fetch(url);

      default:
        return fetch(request);
    }
  }
}

Test each failure mode and observe: Does the CDN fall back to a direct origin fetch when the function throws? Does it respect the function's CPU time limit? Does it return a useful error to the client, or a generic 500? Understanding these failure behaviors is essential for designing resilient edge architectures.

Experiment 4: DNS Resolution Failure

CDN origin resolution depends on DNS. If the CDN's internal DNS resolver can't resolve your origin hostname, all cache misses become failures. This is a surprisingly common production incident — upstream DNS providers have outages, DNSSEC validation fails, or DNS record TTLs expire during a provider migration.

# Simulate DNS failure by changing origin to unresolvable hostname
# Using CDN API to update origin configuration
curl -X PATCH "https://api.cdn-provider.com/v1/zones/${ZONE_ID}/dns_records/${RECORD_ID}" \
  -H "Authorization: Bearer ${API_TOKEN}" \
  -H "Content-Type: application/json" \
  --data '{
    "content": "nonexistent-origin-12345.example.com"
  }'

# Monitor: how long until CDN detects the failure?
# Does it fail open (serve stale) or fail closed (return errors)?
# How quickly does it recover when DNS is restored?

The key finding from DNS chaos experiments is usually about DNS caching behavior. Most CDNs cache DNS resolutions internally with their own TTL, separate from the DNS record's TTL. Understanding this internal caching behavior tells you how long you have to fix a DNS issue before it impacts users.

Building a Chaos Engineering Program

Start with the simplest experiment — origin outage — and run it in a staging environment. Graduate to production only after you've validated your abort mechanisms and established baseline metrics. Build a catalog of experiments that cover your top risk scenarios.

Automated Chaos with Game Days

Schedule regular game days — planned chaos sessions where the team runs experiments and practices incident response. Automate the experiment execution so the team can focus on observation and decision-making, not typing commands.

# game-day-runner.py
experiments = [
    {"name": "origin-outage",  "duration": 300, "blast": "5%"},
    {"name": "cache-purge",    "duration": 120, "blast": "staging"},
    {"name": "latency-inject", "duration": 600, "blast": "10%"},
]

for exp in experiments:
    print(f"\n{'='*60}")
    print(f"EXPERIMENT: {exp['name']}")
    print(f"Duration: {exp['duration']}s | Blast radius: {exp['blast']}")
    print(f"{'='*60}")

    confirm = input("Proceed? [y/N] ")
    if confirm.lower() != 'y':
        continue

    baseline = collect_baseline(duration=60)
    inject_failure(exp['name'], exp['blast'])

    monitor_with_abort(
        duration=exp['duration'],
        baseline=baseline,
        abort_threshold={"error_rate": 0.10, "p99_latency_ms": 5000}
    )

    restore_normal()
    recovery = measure_recovery(timeout=120)

    generate_report(exp, baseline, recovery)

Document findings from every game day. Track which experiments revealed real weaknesses, what was fixed, and whether the fix actually improved resilience in subsequent experiments. This builds institutional knowledge about your CDN's real-world failure modes — knowledge that dashboards and documentation can't provide.

Common Discoveries

• stale-while-revalidate doesn't work as expected — Many CDNs interpret this header differently. Some honor it only for specific cache statuses. Test explicitly to confirm your CDN actually serves stale content during origin failures.
• Health checks are too slow — Default health check intervals (30-60 seconds) mean your CDN continues routing to a failed origin for up to a minute before detecting the failure. Reduce intervals for critical origins.
• Edge functions don't fail open — When an edge function crashes, some CDN providers return a 500 error instead of falling back to the origin. This means a single function bug can take down your entire site.
• Cache purge is not instant — Purge propagation across all edge PoPs can take 5-30 seconds. During this window, some users get cached content while others get fresh content, which can cause consistency issues for stateful applications.
• Origin capacity is lower than expected — After months of CDN caching, teams forget how much traffic the origin actually handles. A full cache purge reveals that the origin can't handle the uncached request rate.