Origin Weighted Failover Patterns: Gradual Traffic Shifting During Outages

Back in late 2019, our primary origin in us-east-1 started shedding connections at 02:14 UTC — not a hard-down, just a slow bleed of 500s and 50ms of extra TTFB creeping up every minute. Our classic active-passive failover flipped 100% of traffic to the us-west-2 origin in under three seconds, and that origin fell over fifteen seconds later. It wasn't ready for a thundering herd with cold caches and empty connection pools. That's when I stopped trusting binary failover.

Why Binary Failover Fails When You Need It Most

Most CDN setups I've audited still use a simple boolean: primary is healthy or it isn't. If health checks fail N times in a row, DNS or the edge router marks the origin down and sends everything to the next in line. It looks clean on a whiteboard, but it creates two brutal problems in production. First, you get a cache stampede. The secondary origin suddenly has to handle 100% of misses with none of the hot objects in memory or on local SSD, so your origin hit ratio drops from 95% to maybe 20% for several minutes. Second, you hammer connection establishments. I've seen secondary origins go from 4k established connections to 80k in under 10 seconds, and if you haven't tuned your Origin Connection Pooling Optimization: Reducing TCP Overhead at CDN Scale properly, you'll exhaust ephemeral ports and file descriptors before your application even gets CPU-bound.

That second failure in 2019 taught us that an origin that's 70% healthy shouldn't be treated the same as one that's 0% healthy. Gradual traffic shifting treats health as a spectrum, not a switch. Instead of 100/0, you move to 80/20, then 50/50, then 20/80, based on continuous signals. You're buying time for caches to warm, autoscaling to react, and connection pools to grow organically. We've since applied this to every critical property, and our incident-driven origin overloads dropped by roughly 60% year-over-year. The logic aligns closely with how the HTTP Working Group Specifications describe retry and failover behavior — you don't want to amplify a partial failure into a total one.

The simplest implementation isn't DNS at all. At the edge, you need a weighted selection that can be updated without a config push. We run an in-memory weight table per POP that our health daemon rewrites every second. If you're using Varnish, Nginx, or Envoy, you can do the same with a shared dictionary or xDS update. Here's a stripped-down version of the Lua we run inside our edge tier to avoid full cutovers:

-- weighted_origin_selector.lua (runs per-request at the edge)
local origins = {
  { name = "primary-ewr", weight = 80, conn_pool = pool_ewr },
  { name = "secondary-iad", weight = 20, conn_pool = pool_iad }
}

-- health_score is 0.0 to 1.0, aggregated from active + passive checks
-- see: /origin-health-monitoring-patterns
function update_weights(health_score)
  -- don't slam the secondary; cap shift to 10 points per second
  local target_primary = math.floor(health_score * 100)
  local current_primary = origins[1].weight
  local delta = math.max(-10, math.min(10, target_primary - current_primary))
  
  origins[1].weight = current_primary + delta
  origins[2].weight = 100 - origins[1].weight
end

function select_origin()
  local r = math.random(100)
  if r <= origins[1].weight then
    return origins[1]
  else
    return origins[2]
  end
end

Designing Linear, Exponential, and Health-Proportional Shifts

Once you've got a weighted selector, the next question is how fast you move the weights. We've tried three patterns in production, and they behave very differently under load. Linear decay is the most predictable: you subtract, say, 10% from the primary every 5 seconds until it reaches zero. It's easy to reason about and easy to alert on, but it's slow to react if the primary suddenly starts returning 100% 5xx. Exponential decay is faster — you halve the primary's weight each interval (80 -> 40 -> 20 -> 10) — so you get away from a truly dead origin quickly while still avoiding an instant 0-to-100 jump on the secondary. Our current default is health-proportional, where the weight directly tracks a composite health score.

That health score is where most teams get it wrong. You can't base it on a single HTTP 200 check to /healthz. We combine active probes (5-second interval synthetic GETs for cacheable and uncacheable paths), passive observations (upstream response time, 5xx rate, and connection errors over a 30-second rolling window), and saturation signals (p95 origin queue time). Each signal produces a 0-1 score, and we take the weighted minimum — not the average — so one failing signal can pull the whole score down. If you're still relying only on active probes, I'd strongly recommend reading our deep dive on Origin Health Monitoring Patterns: Active Probes, Passive Checks, and Hybrid Approaches. Without passive checks, we wouldn't have caught that slow-bleed incident at all; the /healthz endpoint kept returning 200 while real user paths were timing out.

Here's how we translate that composite score into a smoothed weight update with exponential decay and jitter to prevent synchronized thundering herds across POPs. This is the controller loop that runs every second per POP:

// origin_weight_controller.go
type HealthState struct {
    Score       float64   // 0.0 (dead) to 1.0 (perfect)
    EWMA5xx     float64
    P95Latency  float64
}

func computeTargetWeight(state HealthState) int {
    // latency penalty: +50ms over baseline = -20 weight points
    latencyPenalty := math.Max(0, (state.P95Latency - 120) * 0.4)
    penaltyScore := state.Score*100 - latencyPenalty - (state.EWMA5xx*50)
    target := int(math.Max(0, math.Min(100, penaltyScore)))
    return target
}

func reconcileWeights(current, target int) int {
    // exponential approach with per-POP jitter (+/- 2%)
    jitter := rand.Intn(5) - 2
    maxStep := 15 // never shift more than 15% per second
    delta := target - current
    if delta > maxStep { delta = maxStep }
    if delta < -maxStep { delta = -maxStep }
    return current + delta + jitter
}

Tuning Shift Velocity Against Cache Warmth and Connection Churn

You can't tune shift velocity in isolation. The speed at which you move traffic has to match how quickly your secondary can actually absorb it. Two factors dominate: cache warmth and connection pool readiness. If your secondary POP has to re-fetch 80% of objects from origin because its cache is cold, even a 50% traffic shift can double origin load. We learned to model this by looking at cache hit ratio vs. time after a shift. For our image-heavy properties, hit ratio recovers in about 90 seconds; for long-tail API responses that are uncacheable, it never recovers and you're purely constrained by origin CPU and database connections. That recovery window should dictate your ramp duration. Guidance from web.dev Articles on stale-while-revalidate and cache-control tuning is directly applicable here — a longer s-maxage and a well-configured stale-while-revalidate window gives you a much wider safety net during the shift.

The other hidden cost is retries. A gradual shift without a retry budget is just a slower way to overload the secondary. Every request that fails on the primary and is retried on the secondary counts as extra load you didn't plan for in your weight calculation. We enforce a token-bucket retry budget (max 15% of total requests per second can be retried) and use exponential backoff with jitter between retries. If you haven't implemented this yet, our notes on Origin Retry Strategies for CDN Resilience: Backoff, Jitter, and Budget Limits cover the exact algorithm we use to keep retries from amplifying an outage. Without a budget, your 80/20 weight split can easily become 80/35 in real origin QPS once retries are included.

Implementing Weighted Failover in Production: From Health Checks to Edge Config

Most CDNs still default to binary failover — primary is 100% healthy until it isn't, then you slam 100% of traffic to the secondary. To get a gradual shift, you'll need to build a control loop on top of weighted origin groups. We've had good luck treating origins as a weighted pool where the edge or the load balancer adjusts weights every 15-20 seconds based on a sliding window, not an instantaneous health check. Don't tie your shift directly to a single 5xx or timeout; you'll flap on every deploy hiccup.

In practice, that loop looks like this: active health checks every 5 seconds from each PoP with a 2-second connect timeout and 4-second read timeout, plus passive monitoring of real traffic error rates over a 30-second rolling window. When the passive 5xx rate crosses 2% or p95 origin latency spikes past 800ms for two consecutive windows, we start stepping weight off the primary — typically 10% per interval. That gives you a full drain in about 3 minutes, which is slow enough to keep the secondary's cache and connection pools from melting, but fast enough to protect users. If you're not tuning those timeouts tightly, you'll shift too late; our Origin Timeout Configuration Guide: Connect, Read, and Total Timeout Tuning breaks down why that 2s/4s split matters more than a single 10s total timeout.

You'll also want to decouple your hedging from your failover weights. During a shift, we run hedged requests at about 5% duplication to the warming secondary — not full Origin Request Hedging: Sending Parallel Requests to Cut Tail Latency — just enough to prime its cache and confirm it can actually handle the load before it takes 50%+ of traffic. And don't forget TLS. If your secondary origin hasn't been hit in hours, a stale or mismatched certificate will turn your graceful failover into a hard outage. Enforce strict checks on both origins with Origin SSL Verification at CDN Edges: Preventing Man-in-the-Middle Attacks so you don't discover an expired cert at minute two of an incident.

// Weighted gradual failover - CDN origin group config (JSON / Terraform style)
{
  "origin_group": "api-primary-secondary",
  "origins": [
    {
      "id": "primary-ewr",
      "address": "origin-primary.example.com",
      "weight": 100, // dynamically adjusted 100 -> 0
      "health_check": {
        "interval": "5s",
        "timeout": "2s",
        "path": "/healthz",
        "expected_status": 200
      }
    },
    {
      "id": "secondary-sjc",
      "address": "origin-secondary.example.com",
      "weight": 0, // dynamically adjusted 0 -> 100
      "health_check": {
        "interval": "5s",
        "timeout": "2s",
        "path": "/healthz"
      }
    }
  ],
  "failover_policy": {
    "type": "health_proportional",
    "error_threshold": "2%_5xx_over_30s",
    "latency_threshold": "p95 > 800ms",
    "shift_step": 10,          // percent per interval
    "shift_interval": "20s",   // allows cache & connections to warm
    "hysteresis": "5%",        // prevents flapping
    "min_warm_pool": 5         // keep 5% traffic on standby origin always
  }
}

Advanced Patterns and Troubleshooting: When Gradual Shifts Stall or Flap

Linear shifts are easy to reason about, but they're not always what you want. If your secondary is cold — say, cache hit ratio is normally 92% on the primary and the secondary is sitting at 0% because it only gets health checks — a linear 10% step can still double its origin load every 20 seconds. In that case, we've seen better results with an exponential backoff-style shift: 5% → 10% → 20% → 40%, paired with deliberate cache pre-warming. We keep 5-10% of cacheable traffic pinned to the secondary at all times, even when it's "standby," so its hot set never fully evicts. Without that, expect a 15-30% hit ratio drop and a 2-3x surge in origin fetches for the first 90 seconds of the shift.

The other killer is session affinity and false recovery. If your app uses sticky sessions or JWTs pinned to a region, a naive weighted drain will break in-flight users. Don't just shift new connections; drain existing ones with a 60-90 second connection close window and honor `Connection: close` from the origin. And watch for flapping. An origin that recovers for 20 seconds then throws 500s again will yo-yo your weights if you fail back immediately. We require three consecutive healthy windows (90 seconds total) plus p50 latency back under 250ms before we shift weight *back* at half the speed — 5% per 20 seconds. It's intentionally asymmetric: fast to leave, slow to return.

When a shift stalls — weight stuck at 60/40 and error rates aren't improving — check three things first. One, your health check is too shallow. A 200 OK on `/healthz` doesn't mean the API can serve `/v2/checkout`. Use synthetic transaction checks or at least verify downstream dependency status in the payload. Two, you've got origin connection limits tripping before your error percentage does. Monitor active origin connections and queue depth, not just HTTP status. Three, you're shifting weight globally when the failure is regional. If only your EWR PoPs see timeouts to the primary, don't shift SJC traffic. Weighted failover should be PoP-aware; a global shift for a regional blip just creates a second outage.

Frequently Asked Questions