CDN Failover and Origin Redundancy: Ensuring Availability During Outages

When an origin goes dark at 2 AM, the CDN is the only thing standing between your users and a hard error. I've spent the last decade debugging failover events where everything looked correctly configured on paper, yet the edge still served 502s for minutes before traffic shifted. The gap is almost always between the documented behavior of origin redundancy and how it actually operates under load, with real health checks, real connection pools, and real cache states. This article breaks down the mechanics of CDN failover and origin redundancy as they work in production — not in diagrams — so you can design for continuous availability when one or more origins become unreachable.

How Active-Passive Origin Groups Actually Behave Under Failure

In my experience, most teams assume failover is instantaneous: primary fails health checks, traffic moves to secondary. In practice, there is a window where the CDN still has in-flight requests, pooled keep-alive connections, and sometimes even queued retries all pointed at the failing origin. Understanding that window is the difference between a seamless shift and a burst of 5xx errors visible to users.

Most CDN platforms implement origin groups with a priority or rank. A typical setup is one primary and one or two secondaries, but you can also configure active-active with load-based distribution. For high availability, I strongly prefer active-passive for dynamic, uncacheable traffic and active-active only for stateless, cacheable workloads where slight inconsistency during convergence is acceptable. The reason is determinism: with active-passive, you know exactly which origin should receive traffic at any given health state.

What Happens Between Health Check Failure and Traffic Shift

Here is the sequence I've observed on Fastly, CloudFront, Akamai, and Cloudflare Enterprise when a primary origin fails:

First, edge POPs continue to use existing TCP connections until the health check subsystem marks the origin as unhealthy. Health checks run out-of-band from a smaller set of checkers, not from every edge node. That means an edge node in Singapore might still try the primary for 5-15 seconds after the checker in Ashburn has already flagged it as down. Second, once the origin is marked unhealthy, the edge does not immediately close keep-alive pools — it stops selecting that origin for new requests. Requests already forwarded will still wait for an origin response or timeout.

This is why your origin timeout and retry configuration matters as much as the health check itself. If your origin read timeout is 10 seconds and your health check interval is 5 seconds with a threshold of 3 failures, users could see 15-25 seconds of failures before any failover is effective at the edge that serves them.

Connection Pool Draining and Retry Semantics

Effective cdn failover requires you to tune three values together: connect timeout, first-byte timeout, and retry budget. I keep connect timeouts short at the edge — 2 to 3 seconds maximum. If an origin cannot accept a TCP connection in that window, it is not going to serve the request. First-byte timeouts can be longer for dynamic APIs, but you should still cap them at 5-7 seconds for most web traffic.

Retries are where redundancy can hurt you. An unconditional retry on 502/503/504 to the next origin effectively doubles your origin load during a partial outage. If your secondary origin is smaller than your primary, a retry storm can take it down as well. I've found that limiting retries to idempotent methods (GET, HEAD, OPTIONS) and adding a circuit breaker after one retry per request prevents cascading failure.

# Example: VCL logic for safe origin retry (Fastly / Varnish-style)
sub vcl_fetch {
  # Only retry on idempotent requests and specific failures
  if (beresp.status == 502 || beresp.status == 503 || beresp.status == 504) {
    if (req.method == "GET" || req.method == "HEAD") {
      if (req.restarts == 0) {
        # Mark primary as sick for this POP briefly
        set beresp.saintmode = 10s;
        return(restart);
      }
    }
  }
  # For stale-if-error, keep stale object for 1 hour on failure
  if (beresp.status >= 500 && stale.exists) {
    return(deliver);
  }
}

The pattern above uses saint mode (or its equivalent) to quarantine the failing origin at the POP level for 10 seconds after a single failure, which allows faster local failover than waiting for global health check convergence. Combined with proper health checks, this gives you two layers of protection: fast local reaction and durable global shift.

Health Checks That Don't Lie: Designing Probes for True Origin Availability

A health check that returns 200 OK from a load balancer is not a health check. I have seen origins marked healthy for hours while the application behind the load balancer was returning 500s for every real request because the probe hit /health which only checked that Nginx was running. For origin redundancy to be meaningful, your probe must validate the full path a user request would take.

Beyond 200 OK: Validating Application Health

A useful origin probe should do three things: hit the same host header and path prefix that production traffic uses, validate response body content, and traverse critical dependencies. For example, if your application needs database and cache to render pages, your health endpoint should execute a lightweight read from both. A simple static file check will not catch a database failover that leaves your app returning error pages.

I recommend creating a dedicated health endpoint like /cdn-health that is not cached at the edge, performs a shallow dependency check, and returns a JSON payload you can match against. Configure the CDN to expect both a 200 status and a specific string, such as "status":"ok". This prevents a misconfigured origin that returns a 200 error page from being considered healthy.

// Example: Cloudflare Worker style health check validation
// Origin must return 200 and JSON with { "status": "ok", "region": "us-east-1" }
async function isOriginHealthy(originUrl) {
  const controller = new AbortController();
  const timeout = setTimeout(() => controller.abort(), 3000);
  try {
    const res = await fetch(originUrl + "/cdn-health", {
      signal: controller.signal,
      headers: { "Host": "www.example.com", "User-Agent": "CDN-HealthCheck/1.0" },
      cf: { cacheTtl: 0, cacheEverything: false }
    });
    if (res.status !== 200) return false;
    const body = await res.json();
    return body.status === "ok" && body.db === "connected";
  } catch (e) {
    return false;
  } finally {
    clearTimeout(timeout);
  }
}

Tuning Intervals, Timeouts, and Thresholds to Avoid False Positives

Aggressive health checks detect failures faster but increase origin load and cause flapping on transient blips. Conservative checks are stable but slow to fail over. In production, I use an interval of 5 seconds, a timeout of 3 seconds, 3 consecutive failures to mark unhealthy, and 2 consecutive successes to mark healthy again. This gives a detection time of ~15 seconds and recovery time of ~10 seconds, which balances responsiveness with stability.

For high availability deployments with multiple origins across regions, stagger your health checks so you are not probing all origins from all checkers at the exact same second. Most CDNs do this automatically, but if you run your own checker (for DNS-based failover), add jitter of 10-20% to the interval.

I've found that body-match checks hit the sweet spot for most teams. Synthetic transactions are valuable for monitoring but are usually too heavy to run every 5 seconds from every POP against every origin. Reserve them for your external uptime monitoring and keep CDN health checks focused and lightweight.

Failover at the Edge vs Failover at DNS: Where Decisions Should Live

There are two fundamentally different places to make a failover decision: at the DNS layer before the user ever connects to an edge, and at the edge after the request has arrived. Both have a role in origin redundancy, but they operate on different timescales and have different consistency guarantees.

DNS-based failover typically relies on Global Server Load Balancing: GSLB Strategies for Multi-Region Deployments to move users away from an entire CDN region or origin region by changing the answer returned for your hostname. This is powerful for steering traffic away from a failed region, but it is limited by DNS TTL and resolver caching. Even with a 30-second TTL, I've measured that 15-25% of resolvers will continue to use a stale answer for several minutes. DNS failover is a coarse, slow tool.

Edge-based failover, by contrast, happens inside the CDN after the request has been routed to a POP. The edge has already terminated TLS, has cache state, and can instantly retry against a different origin without any client-visible DNS change. This is where you want your primary failover logic for origin redundancy. DNS should be your second line of defense for when an entire edge region or CDN provider is impaired.

When to Use Each Layer

Use edge failover for: single origin instance failure, availability zone failure within a region, and deployment rollbacks where you need immediate shift for uncacheable requests. It is fast, has no TTL dependency, and can respect request method and cache state.

Use DNS/GSLB failover for: full regional outage where all origins in a region are down, disasters where the CDN itself cannot reach the region, and for active-active multi-CDN strategies where you need to shift user-to-edge assignment. Pairing edge failover with Anycast DNS Architecture: How CDNs Route Users to the Nearest Edge gives you both fast origin switching and fast edge steering without relying solely on TTL expiration.

In practice, I configure both. The edge handles 95% of failover events in seconds. GSLB handles the remaining 5% of catastrophic events where the edge cannot reach any origin in its preferred region and needs to send the user to a different geography entirely.

Shield Tiers and Regional Origins: Containing Failover Blast Radius

Without an origin shield, every edge POP that experiences a cache miss will directly contact your origin. During a failover, this can amplify into a thundering herd against your secondary origin. Origin shielding collapses those requests into a single tier — typically one or two shield POPs per region — that fetch from origin and then fan out to the edges.

This topology changes your redundancy model. Instead of N edges -> 2 origins, you have N edges -> 2 shield POPs -> 2 origins. Failover now happens at the shield layer first. If your primary shield cannot reach the primary origin, it tries the secondary before returning an error to the edges that requested the object. This containment is critical because it limits the number of concurrent origin connections during an outage to the number of shields, not the number of edges.

I recommend mapping shield regions to origin regions explicitly. For example, tie the US-EAST shield to your us-east-1 origin as primary and eu-west-1 as secondary, and tie the EU shield to eu-west-1 as primary and us-east-1 as secondary. This keeps failover traffic local when possible and only crosses the ocean when a whole region is down. Most major CDNs allow you to configure shield-to-origin affiliation per origin group.

Collapsing Forwarding and Request Coalescing

Enable request collapsing (also called request coalescing or collapsed forwarding) at the shield. When 100 edge nodes request the same uncached object simultaneously, the shield should forward only one request to origin and hold the other 99 pending until the response arrives. If that single origin request fails and triggers a failover to the secondary origin, you have avoided sending 100 failed requests to the secondary. I've seen secondary origins survive failover events specifically because collapsing was enabled, and fail when it was not.

Test this behavior under load before you need it. Generate a cache miss storm against a single URL and then take the primary origin down mid-test. Observe whether the shield retries once or per edge request. The difference determines whether your secondary needs to handle 1x or 100x load during the first seconds of an outage.

Stale-While-Revalidate and Stale-If-Error: Serving During Origin Blackouts

Even with perfect failover, there will be moments when all origins are unreachable or when failover itself takes longer than your users' patience. The most effective high availability technique I have used is not faster failover — it is not needing an origin at all for a period of time.

HTTP's Cache-Control extensions stale-while-revalidate and stale-if-error, defined in the HTTP Working Group Specifications, allow a CDN to serve stale cached content while it tries to revalidate in the background, or specifically when the origin returns an error. This is documented in detail on HTTP Working Group Specifications (https://httpwg.org/specs/) and explained with practical guidance on web.dev Articles (https://web.dev/articles/). When your origins return 500s for 60 seconds, users can still receive content that is 60 seconds stale rather than an error page.

# Origin response headers for resilient caching
# Cache for 60 seconds, serve stale for 5 minutes while revalidating,
# and serve stale for 1 hour if origin errors
Cache-Control: public, max-age=60, stale-while-revalidate=300, stale-if-error=3600
CDN-Cache-Control: public, max-age=300
Surrogate-Control: max-age=86400, stale-while-revalidate=86400, stale-if-error=86400

In my deployments, I set browser-facing max-age short (60s) while allowing the CDN to keep and serve stale for much longer via CDN-Cache-Control or Surrogate-Control. This way browsers revalidate frequently for freshness, but the edge can absorb an origin outage for hours if necessary. For HTML documents where staleness is more sensitive, I use shorter stale-if-error windows (60-300 seconds) and combine it with a lightweight offline shell.

Implementing Stale Serving at the Edge

Not all CDNs honor stale-if-error by default — you often must explicitly enable it. On Varnish/Fastly, this is a VCL setting. On Cloudflare, it requires an origin rule or Worker. On Akamai, it is the serve-stale behavior. Verify that your platform will serve stale on 502/503/504/520-530 and also on origin timeouts, not just on explicit 500 responses. Timeouts are the most common failure mode during network partitions and are sometimes handled differently than HTTP error statuses.

One caveat: stale-while-revalidate for personalized or authenticated content must be carefully scoped. Never serve stale personalized content to the wrong user based on a collapsed cache key. Use Vary correctly or bypass stale serving entirely for private content by sending Cache-Control: private, no-store from origin.

Automating Origin Quarantine and Reintroduction Without Flapping

Manual failover is not failover — it is an incident response runbook. True origin redundancy requires automation that quarantines a failing origin, shifts traffic, and then reintroduces it gradually without flapping. Flapping is when an origin recovers enough to pass one or two health checks, gets traffic back, immediately fails under load, and is removed again, repeating indefinitely. This oscillation is worse than staying on the secondary because it creates intermittent errors for a subset of users.

I implement quarantine with a failure count and a cool-down period. After an origin is marked unhealthy (3 failures), it stays out of rotation for a minimum of 60 seconds regardless of health check results. Reintroduction is not all-at-once; the CDN should bleed a small percentage of traffic back to the recovered origin — 1%, then 10%, then 50% — before returning it to 100%. This is where Traffic Splitting for Progressive Rollouts: Weighted Routing at the CDN Layer concepts apply directly to failover recovery.

# Terraform example: Origin group with health checks and gradual recovery
resource "cdn_origin_group" "primary_group" {
  name = "app-origin-group"

  origin {
    name       = "primary-us-east"
    address    = "origin-primary.example.com"
    priority   = 1
    weight     = 100
  }

  origin {
    name       = "secondary-eu-west"
    address    = "origin-secondary.example.com"
    priority   = 2
    weight     = 100
  }

  healthcheck {
    path                = "/cdn-health"
    expected_status     = 200
    expected_body       = "\"status\":\"ok\""
    interval            = 5
    timeout             = 3
    unhealthy_threshold = 3
    healthy_threshold   = 2
    # Hold unhealthy origin out for 60s before re-probing for recovery
    quarantine_time     = 60
  }

  failover_policy {
    max_retries              = 1
    retry_condition          = "502,503,504,timeout"
    retry_only_idempotent    = true
    # Gradual reintroduction: 5% -> 25% -> 100% over 3 minutes
    recovery_steps           = [5, 25, 100]
    recovery_interval        = 60
  }
}

For reintroduction, I also use synthetic canary requests from the shield rather than real user traffic. The shield sends a background request to the quarantined origin every 15 seconds. Only when two consecutive canaries succeed does it begin the weighted ramp. This prevents user-facing errors during the probe phase. If you do not have canary support in your CDN, approximate it by setting a much higher healthy threshold (5-7 successes) before re-adding the origin.

Observability You Cannot Skip

You cannot automate what you cannot see. For every origin group, monitor at minimum: origin health state per POP, origin request volume and error rate per origin, retry rate, stale serve rate, and time-to-failover measured from first failed health check to last request sent to the failed origin. Alert on retry rate exceeding 10% — it often indicates a partial outage that has not yet tripped health check thresholds but is already affecting users. I also run a weekly failover drill where I blackhole the primary origin with a firewall rule for 5 minutes and measure the error budget impact. In my experience, teams that drill monthly catch configuration drift that would have caused real outages.

Frequently Asked Questions