Origin Disaster Recovery Planning for CDN: Multi-Region Failover Strategies

It was 2:17 AM on a Tuesday when our primary origin in us-east-1 went completely dark — not a graceful degradation, but a full network partition that took our API and our image pipeline with it. The CDN kept serving stale content for about 90 seconds, and then the error rate shot to 43% because we’d never properly tested our failover to eu-west-1. That night taught me that CDN continuity isn’t about having a backup origin on paper; it’s about whether your disaster recovery plan actually works when you’re half-asleep and getting paged.

Why Single-Origin Setups Break Your Disaster Recovery Promise

For years I’ve seen teams treat origin DR as an afterthought. They’ll spend months tuning cache hit ratios and edge logic, then point every POP on the planet at a single origin cluster behind one load balancer. It works until it doesn’t. A regional outage, a bad deploy that passes health checks but returns 500s for real users, or even a fiber cut between your CDN provider and your cloud region can turn that single origin into a single point of failure for your entire global footprint. I’ve audited setups where the “DR plan” was literally “DNS will fail over” — without any thought given to TTLs, connection draining, or how the CDN handles origin stickiness.

The core problem is that CDNs are optimized to maintain persistent, efficient connections to origins. That efficiency becomes a liability during an outage if you haven’t planned for it. Without explicit Origin Connection Pooling Optimization: Reducing TCP Overhead at CDN Scale paired with a failover-aware strategy, your edge nodes will keep hammering a dead origin, queuing requests, and timing out instead of quickly shifting traffic. You need health signals that are faster than your users’ patience and routing logic that trusts those signals. The HTTP Working Group Specifications are pretty clear on how retries and stale-while-revalidate should behave, but most CDN configs I review don’t implement them correctly for multi-origin cases.

Here’s the pattern I now use as a baseline for detecting a sick origin and shedding it quickly. It’s not fancy, but it’s saved us twice in the last three years. This is a simplified edge routing snippet similar to what we run on our reverse proxy layer in front of origins:

# Origin failover logic - edge health-aware routing
origin_group "primary_and_dr" {
  primary = "origin-us-east-1.example.com"
  secondary = "origin-eu-west-1.example.com"
  
  health_check {
    path = "/healthz/deep"
    interval = 5s
    timeout = 2s
    healthy_threshold = 2
    unhealthy_threshold = 3
    # Check for 200 AND valid JSON body, not just TCP connect
    expected_body = "\"status\":\"ok\""
  }

  failover_conditions {
    on_status = [500, 502, 503, 504]
    on_timeout = true
    on_circuit_open = true
    # Don't fail over on 429/403 - that's not an origin outage
  }

  # Shed primary after 3 consecutive failures within 10s
  circuit_breaker {
    failure_threshold = 3
    window = 10s
    cooldown = 30s
  }
}

Choosing Your Multi-Region Failover Model: Active-Passive vs. Active-Active

There’s no one-size-fits-all answer here, and anyone who tells you otherwise hasn’t operated both. In an active-passive model, all CDN traffic hits your primary region (say, us-east-1) and only shifts to the secondary when the primary is declared unhealthy. It’s simpler to reason about data consistency — you don’t have to worry about write conflicts or replication lag — and it’s generally cheaper since your DR region can run at a smaller capacity. I’ve recommended this for content-heavy sites where origins are mostly read-only and replication is asynchronous. The downside is that your multi-region failover is cold. If you haven’t done regular failover drills, you’ll discover that your secondary origin can’t handle full load only when you need it most.

Active-active is what we moved to after that 2 AM incident. Both regions serve traffic all the time, either via geo-routing (US users to US origin, EU users to EU origin) or via latency-based routing. When one region fails, the other simply absorbs the load. Your disaster recovery becomes a capacity planning problem, not a cold-start problem. You’ll need to keep replication tight and handle conflict resolution, but your CDN continuity during an outage is dramatically smoother. Users don’t see a 30-second blip while DNS propagates; they just get routed to the next healthy origin in the pool. This closely ties into how you handle retries — a well-tuned Origin Retry Strategies for CDN Resilience: Backoff, Jitter, and Budget Limits policy prevents the retry storm that can take down your remaining healthy origin when the first one fails.

If you’re on a cloud provider, don’t try to hand-roll this with DNS alone. Use infrastructure as code so your failover group is versioned and testable. This Terraform snippet creates a proper health-checked origin group for a CDN distribution with automatic failover:

resource "aws_cloudfront_origin_group" "multi_region_dr" {
  origin_id = "multi-region-failover-group"

  failover_criteria {
    status_codes = [500, 502, 503, 504]
  }

  member {
    origin_id = aws_cloudfront_origin.primary_us_east.id
  }

  member {
    origin_id = aws_cloudfront_origin.secondary_eu_west.id
  }
}

resource "aws_route53_health_check" "origin_deep_check" {
  fqdn              = "origin-us-east-1.example.com"
  port              = 443
  type              = "HTTPS"
  resource_path     = "/healthz/deep"
  failure_threshold = 3
  request_interval  = 10

  # Measure latency and status - not just TCP
  measure_latency = true
  invert_healthcheck = false
}

resource "aws_cloudfront_origin" "primary_us_east" {
  domain_name = "origin-us-east-1.example.com"
  origin_id   = "primary-us-east"

  custom_origin_config {
    http_port              = 80
    https_port             = 443
    origin_protocol_policy = "https-only"
    origin_read_timeout    = 10  # Don't wait 30s to declare failure
    origin_keepalive_timeout = 5
  }
  # Connection pooling tuned for fast failover
  connection_attempts = 2
  connection_timeout  = 3
}

What Actually Triggers Failover: Health Signals You Can Trust

The worst failover systems I’ve debugged were the ones that relied on a single signal, usually a shallow HTTP 200 from “/”. Your origin can return 200 OK on “/” while your database is down, your cache is corrupted, or your app is serving 500s for 30% of product pages. For origin DR to be reliable, you need layered health checks. I now insist on three signals: active probes from the CDN edge (synthetic requests every 5 seconds to a deep health endpoint that checks database, cache, and critical dependencies), passive checks (real user request failure rates, latency p95, and origin 5xx ratio observed by the edge), and a hybrid arbiter that correlates both. Our deep health endpoint does a real read from our primary datastore and verifies we can generate a signed URL, because that’s what actually breaks for users.

This is where most teams forget about the front-end impact. Even with perfect origin failover, if your cache headers are wrong, users will still see broken pages. We set stale-while-revalidate and stale-if-error aggressively on cacheable content, so the edge can serve stale for 60-300 seconds while the failover completes. The guidance in web.dev Articles on caching best practices is a good refresher here, especially for handling immutable assets versus personalized API responses. And don’t forget to monitor the failover itself — if you aren’t alerting on Origin Health Monitoring Patterns: Active Probes, Passive Checks, and Hybrid Approaches state changes, you won’t know you’ve failed over until customers tell you.

Implementing Failover at the Edge: Wiring Up Origin Groups Without Downtime

Here's where most teams mess it up — they'll configure two origins in the CDN console and assume they're protected. They're not. A proper origin group needs explicit priorities, health check integration, and timeout behavior that won't cascade into a full edge collapse. You can't just add a backup IP and call it DR.

In production, we define origin groups with a primary and one or two secondaries, each with its own health check and failover threshold. Your CDN should probe each origin every 10-30 seconds (we've found 15 seconds is the sweet spot between detection speed and origin load), requiring 3 consecutive failures before marking unhealthy and 2 consecutive successes to bring it back. That prevents a single dropped packet from triggering a region-wide flip. Crucially, your failover logic must live at the edge PoP, not centrally — if your control plane goes down, your data plane should still route.

Don't forget to tune your timeouts per-origin. A 2-second connect timeout and 5-second read timeout for your primary is aggressive but prevents users from waiting 30 seconds for a dead origin. For secondaries, you can be slightly more lenient, but never exceed 10 seconds total. For a full breakdown of how these values interact, see our Origin Timeout Configuration Guide: Connect, Read, and Total Timeout Tuning. And if you're using HTTPS to origin (and you should be), verify your failover target handles SNI and certificate validation correctly — otherwise you'll fail over straight into a TLS handshake error. We cover that in Origin SSL Verification at CDN Edges: Preventing Man-in-the-Middle Attacks.

# Example: Terraform-style origin group with health-aware failover
resource "cdn_origin_group" "primary_dr" {
  name = "app-origin-dr-group"
  
  origin {
    host     = "origin-us-east-1.app.example.com"
    priority = 1
    # Primary handles 100% of traffic when healthy
    health_check {
      path                = "/healthz"
      interval_seconds    = 15
      timeout_seconds     = 3
      healthy_threshold   = 2
      unhealthy_threshold = 3
      expected_codes      = [200, 204]
    }
  }

  origin {
    host     = "origin-eu-west-1.app.example.com"
    priority = 2
    health_check {
      path                = "/healthz"
      interval_seconds    = 15
      timeout_seconds     = 3
      healthy_threshold   = 2
      unhealthy_threshold = 3
    }
  }

  failover_criteria {
    status_codes          = [500, 502, 503, 504]
    connect_timeout       = "2s"
    read_timeout          = "5s"
    max_retries           = 1  # Don't hammer a dying origin
  }
}

When Failover Fails: Flapping, Split-Brain, and the Thundering Herd

You've got failover configured — great. Now it flaps every 4 minutes and takes your cache hit ratio from 94% to 38%. This is the most common post-launch incident we see. It happens when your health check is too sensitive or your origin is marginally overloaded. If your origin's p95 response time is 1.8s and your health check timeout is 2s, you'll get intermittent failures that trigger failover, which then shifts 12 Gbps of traffic to your secondary, which then also keels over. We've seen this cascade take down both regions in under 90 seconds.

Fix it with hysteresis and tiered load shedding. First, use different thresholds for failing vs. recovering — require 30-45 seconds of sustained failure before failover, but 60-90 seconds of clean health before failback. Second, never cold-switch 100% of traffic at once. Use gradual draining or Origin Request Hedging: Sending Parallel Requests to Cut Tail Latency for the first 1-2% of requests: send a hedged request to the secondary 300ms after the primary, take whichever responds first. It cuts tail latency by 40-60% during degraded states without slamming your standby origin. And always serve stale on failover — a 5-minute-old cached response with `stale-while-revalidate` and `stale-if-error` is infinitely better than a 503.

Another killer is split-brain DNS or Anycast misrouting. If your health checks pass in us-east but fail from eu-central PoPs, some edges will see primary as healthy while others don't. You'll get inconsistent content and cache fragmentation. Solve this by aggregating health from multiple edge regions before marking an origin down, and log every failover event with PoP, origin, and trigger reason. If you can't answer "which PoPs failed over at 14:03 UTC and why?" in under 2 minutes, your observability isn't DR-ready.

Frequently Asked Questions