CDN Disaster Recovery Planning: Failover, Backup Origins, and RTO/RPO

Most teams over-invest in CDN performance tuning and under-invest in recovery. Yet the cost of a 30-minute origin outage during peak traffic can exceed a year of CDN spend. Effective CDN fundamentals must include resilience: redundant origins, intelligent failover, and cache behavior that preserves availability even when your primary infrastructure is unreachable.

Disaster recovery for a CDN is not just “add a second origin.” It is a system that spans DNS, edge logic, origin health detection, cache control, and deployment automation. Without clear Recovery Time Objective (RTO) and Recovery Point Objective (RPO) definitions, you cannot prioritize what to protect or how fast to fail over.

1. Why RTO and RPO Matter for CDNs

RTO is how long your content can be unavailable before business impact becomes unacceptable. RPO is how much data or content freshness you can afford to lose. For CDNs, RPO often translates to cache staleness: how old can a served object be before it is considered incorrect?

Static assets like images and JavaScript have a relaxed RPO — serving a 2-hour-old logo is fine. Personalized API responses, pricing, or inventory have a strict RPO measured in seconds. Your DR plan must tier content accordingly. A single global RTO/RPO will either be too expensive or too lax.

Define these tiers with product and engineering stakeholders, then map them to cache-control headers and origin groups. This prevents the common mistake of treating all traffic as Tier 0, which drives unnecessary cost and complexity. For guidance on balancing performance and freshness, review CDN caching strategies and Core Web Vitals optimization together with your DR tiers.

2. Failover Architectures That Actually Work

There are three failover layers to coordinate. Fail at any one and the others cannot compensate.

The most resilient pattern is edge-driven failover combined with origin shielding. The edge continuously probes /healthz on both origins. On 3 consecutive failures or p95 latency > 2s, traffic shifts automatically. Shield ensures only one POP fetches from backup, protecting it from overload.

3. Backup Origins: Active-Passive vs Active-Active

Your backup origin is not a cold S3 bucket you update manually. It must be deployable, testable, and warm.

Active-Passive (Recommended for Most Teams)

Primary serves 100% of traffic. Backup is synced via replication (database, object storage) and auto-scaling is warm but scaled to zero or minimal instances. CDN sends traffic only on failure. This is simpler, cheaper, and avoids data divergence. Ideal when RPO allows 30-60 seconds of replication lag.

Active-Active (For Tier 0 Workloads)

Both origins serve traffic via latency-based routing or weighted origin groups (e.g., 90/10). Requires conflict-free replication and idempotent writes. Use when RTO must be near-zero and you cannot tolerate failover warm-up time. It doubles operational complexity, so reserve it for checkout, authentication, and real-time APIs.

In either model, keep backup origins in a different cloud region or provider. A regional outage that takes down primary should not take down backup. Store static fallback content — like a lightweight offline shell — in immutable object storage (S3, GCS, R2) fronted by the CDN so it can be served even if compute origins are gone.

Synchronize content with versioned deploys, not rsync hacks. Your CI/CD should deploy to both origins atomically and verify health before marking the release as stable. And remember, image optimization pipelines should also replicate transformed assets to the backup bucket to avoid broken images during failover.

4. Health Checks, Stale Serving & Shielding

Failover is only as good as failure detection. Poor health checks cause flapping or, worse, no failover at all.

• Probe the right thing: Check /healthz that validates app + DB + cache dependencies, not just TCP port 443. Return 200 only if the origin can actually serve traffic.
• Tune thresholds: 3 failures over 15 seconds with 2-second timeout balances speed vs false positives. Require 3 successes to mark healthy again to avoid flapping.
• Enable stale-if-error: Cache-Control: stale-if-error=86400, stale-while-revalidate=60 lets the edge serve cached content for up to 24 hours if origins return 5xx. For Tier 2 assets this alone meets RTO without any origin.
• Use tiered cache / shield: Shield reduces origin load from thousands of edge POPs to a single shield fetch. During recovery, this prevents backup origin overload.
• Persist cache: Enable edge persistence or durable cache where available so restarts do not wipe stale content needed for DR.

Also protect failover itself from abuse. Attackers often trigger origin failures to force fallback to less-protected infrastructure. Keep WAF, bot management, and rate limiting active on backup paths.

5. Configuration Examples & Automation

Declare failover in code, not console clicks. Here are production-ready patterns:

CloudFront Origin Group (Terraform)

resource "aws_cloudfront_distribution" "cdn" {
  origin_group {
    origin_id = "failover-group"
    failover_criteria { status_codes = [500, 502, 503, 504] }
    member {
      origin_id = "primary-alb"
    }
    member {
      origin_id = "backup-s3"
    }
  }
  origin {
    origin_id   = "primary-alb"
    domain_name = "origin-primary.example.com"
    custom_origin_config {
      http_port = 80
      https_port = 443
      origin_protocol_policy = "https-only"
      origin_read_timeout = 5
    }
  }
  origin {
    origin_id   = "backup-s3"
    domain_name = "failover-bucket.s3.amazonaws.com"
    s3_origin_config { origin_access_identity = aws_cloudfront_origin_access_identity.oai.cloudfront_access_identity_path }
  }
  default_cache_behavior {
    target_origin_id = "failover-group"
    cache_policy_id  = aws_cloudfront_cache_policy.caching_optimized.id
    # Serve stale on error
    response_headers_policy_id = aws_cloudfront_response_headers_policy.stale_if_error.id
  }
}

VCL / Edge Logic for Stale Fallback

sub vcl_fetch {
  # Keep stale for 24h for static assets
  if (beresp.status >= 500 && beresp.status < 600) {
    if (stale.exists) {
      return (deliver_stale);
    }
    # Retry backup origin once
    if (req.restarts == 0) {
      set req.backend_hint = backup_origin;
      return (restart);
    }
  }
  set beresp.stale_if_error = 24h;
  set beresp.stale_while_revalidate = 60s;
}

Health Check Automation

# Synthetic check every 10s from 3 regions
curl -s -o /dev/null -w "%{http_code} %{time_total}\n" \
  https://origin-primary.example.com/healthz \
  -H "User-Agent: cdn-health-check/1.0" --max-time 2

# Alert if >2 failures in 30s, auto-promote backup via API
if [ $fail_count -ge 3 ]; then
  cdn_api update-origin-group --primary-status disabled --notify oncall
fi

Version and audit all CDN configurations. A bad cache rule can be as destructive as an origin outage. Use blue-green config deploys and automatic rollback if error rates spike within 5 minutes of a push.

6. Testing, Runbooks & Continuous Validation

A DR plan that has never been tested is just documentation. Schedule these exercises:

• Monthly: Automated failover drill — block primary origin via edge ACL and verify backup serves within RTO. Measure time-to-detection and time-to-recovery.
• Quarterly: Full region failure simulation — power off primary region, validate replication lag meets RPO, and confirm shield + stale serving handles traffic.
• On every deploy: Synthetic canary that fetches critical pages through CDN with Cache-Control: no-cache to force origin fetch from both origins.

Your runbook should be one page, not twenty. Include: detection signal, who decides to fail over (ideally automated), how to verify backup health, how to fail back safely without cache poisoning, and communication template. Store it where on-call can access it even if your primary docs site is down — ideally in the CDN itself as a static page.

Finally, track RTO/RPO as SLIs. Log origin_failover_duration_seconds and content_staleness_seconds in your observability stack. Review them in post-incident reviews and tie them to error budgets. A CDN that recovers in 20 seconds but serves 10-minute-old prices may still breach Tier 0 RPO.

Disaster recovery is not a one-time project. As you add new content types, edge functions, and cost optimization rules, re-evaluate which tier they belong to and whether your backup origins can handle the load. Resilience is a feature you ship continuously.