CDN Retry Storm Prevention: Stopping Thundering Herds of Retried Requests

Back in late 2021, I watched a 400ms bump in origin latency take down an entire edge tier that should’ve easily absorbed it. A database failover stalled responses for less than half a second, and within two seconds our edge nodes had turned that tiny stutter into 11 million retries per second hammering the already sick origins. It wasn’t a traffic spike - it was us. We’d built the thundering herd ourselves with well-intentioned retry logic.

Why a Single Origin Hiccup Turns Into a Global Retry Storm

Most CDN retry storms don’t start with an outage. They start with perfectly reasonable defaults. You’ve got 200+ PoPs, each running maybe 50-100 edge workers or Nginx threads, all configured to retry once on a 5xx or a timeout. On paper that looks safe. In practice, you’ve just created a massive distributed amplifier. If your origin normally handles 40k RPS and suddenly slows down, even a 5% failure rate means 2,000 failed requests per second. With one blind retry, that’s 2,000 extra requests. But those retries don’t spread evenly - they cluster in the exact moment the origin is least able to handle them.

I’ve seen teams make this worse by setting aggressive timeouts. They’ll drop the origin read timeout to 500ms to “fail fast,” then retry immediately on another connection. What they’re really doing is converting slow responses into hard failures, which then trigger more retries. You haven’t avoided the slow origin; you’ve just turned every slow request into two fast, failing ones. When you’re running at edge scale, that retry multiplier can easily be 3x to 8x if you’ve got multiple layers retrying - the client retries, the edge retries, and maybe your shield layer retries again. That’s how a transient 2% error rate becomes a full-blown retry storm that prevents the origin from ever recovering. Understanding this amplification is core to SRE Practices for CDN: Applying Site Reliability Engineering to Content Delivery, because you can't fix what you don't model as a feedback loop.

The math gets ugly fast because retries are correlated. They aren’t random background noise. Hundreds of edge nodes detect the failure at roughly the same time and all retry in near-perfect synchrony. Without jitter and backoff, you get a wall of traffic that hits the origin exactly when its queue is fullest. The HTTP specifications are pretty clear that retries need to be conditional and idempotent for a reason, and HTTP Working Group Specifications provide the ground rules for when a retry is even safe to attempt. Ignoring those constraints is how you turn a recoverable blip into a sustained outage that lasts long after the database has recovered.

// Naive retry vs. jittered backoff in an Edge Worker (JavaScript)
// DON'T do this at the edge - synchronized retries create a thundering herd

// Bad: immediate, synchronized retry
async function fetchWithNaiveRetry(request) {
  let res = await fetch(request);
  if (res.status >= 500) {
    // Every edge node does this at the same time
    res = await fetch(request);
  }
  return res;
}

// Good: exponential backoff with full jitter + idempotency check
async function fetchWithSafeRetry(request, maxRetries = 2) {
  for (let attempt = 0; attempt <= maxRetries; attempt++) {
    // Only retry safe methods - don't retry POSTs blindly
    if (attempt > 0 && !['GET','HEAD','OPTIONS'].includes(request.method)) {
      break;
    }
    try {
      const controller = new AbortController();
      const timeout = setTimeout(() => controller.abort(), 800 + attempt * 400);
      const res = await fetch(request, { signal: controller.signal });
      clearTimeout(timeout);
      
      if (res.status < 500) return res;
      if (res.headers.get('x-no-retry')) return res; // Origin says don't retry

    } catch (e) {
      if (attempt === maxRetries) throw e;
    }
    // Full jitter: random delay between 0 and (base * 2^attempt)
    const baseMs = 100;
    const capMs = 1500;
    const delay = Math.random() * Math.min(capMs, baseMs * Math.pow(2, attempt));
    await new Promise(r => setTimeout(r, delay));
  }
}

Circuit Breakers and Request Hedging That Don't Make Things Worse

A retry is an optimistic bet that the next attempt will succeed. A circuit breaker is the opposite - it’s a pessimistic admission that the next attempt will probably fail, so you shouldn’t even try. At the CDN edge, you need both, and you need them to coordinate. I learned this the hard way after we added retries to fix 0.1% origin errors and then had to add circuit breakers to fix the retries. The pattern that actually holds under load is pretty simple: track failure rate per origin or per route in a rolling window, and when it trips a threshold, fail fast for a short period instead of proxying the request at all.

The sweet spot I’ve found for most high-traffic properties is a 10-second rolling window with a 50% error threshold over at least 100 requests. If you trip it, open the circuit for 5-15 seconds and serve stale, send a 503 with a Retry-After, or shed load gracefully. Don’t keep the circuit open for 60 seconds - you’ll just create a different kind of herd when it closes and everyone reconnects at once. And don’t implement circuit breakers per-edge-thread; you need shared state via something like a centralized counter in your PoP or via consistent hashing to a shield. Otherwise, each of your 80 workers opens and closes its own circuit out of sync, and you get oscillation. For a deeper look at managing those trade-offs under pressure, see CDN Error Budget Management: Balancing Reliability with Feature Velocity.

Request hedging is the subtler alternative to retries that many teams misuse. Instead of waiting for a timeout and then retrying, you send the same request to a second origin after the 95th percentile latency has passed - say, 300ms - and take whichever returns first. It can dramatically cut tail latency without a full retry storm, but only if you hedge a very small percentage of requests. Google’s research suggests hedging more than 2-5% of traffic at peak actually increases load. At the edge, we’ll typically only hedge GETs for cache misses, and we strictly cap it to one hedged request per original. We also use a budget: no more than 10% of current origin concurrency can be hedged requests. Articles on web.dev Articles show how this impacts user-perceived latency, but from the CDN stability side, unbounded hedging is just a retry storm with a different name. Automating these safeguards so no one can deploy a naive retry config is exactly what we cover in CDN Toil Reduction Through Automation: Eliminating Repetitive Operations Work.

# Envoy-style circuit breaker + retry budget for CDN origin cluster
# Applied at the edge proxy / shield layer - NOT per client

clusters:
- name: api_origin
  connect_timeout: 0.5s
  circuit_breakers:
    thresholds:
      # Stop creating new connections when origin is saturated
      max_connections: 6000
      max_pending_requests: 4000
      max_requests: 8000
      max_retries: 800          # Hard cap: retries can't exceed 10% of max_requests
      track_remaining: true
  outlier_detection:
    consecutive_5xx: 5
    interval: 5s
    base_ejection_time: 15s
    max_ejection_percent: 50    # Don't eject all hosts at once
    failure_percentage_threshold: 50
    failure_percentage_minimum_hosts: 3

retry_policy:
  retry_on: "5xx,reset,connect-failure"
  num_retries: 1                # Never more than 1 retry at the edge
  per_try_timeout: 1.2s
  retry_back_off:
    base_interval: 0.1s
    max_interval: 1.5s
  # Critical: enforce a retry budget
  retry_host_predicate:
  - name: envoy.retry_host_predicates.previous_hosts
  host_selection_retry_max_attempts: 2

How to Spot Retry Pressure Before It Flattens Your Origin

You can’t prevent retry storms with just good configs; you need signals that tell you pressure is building before the origin’s CPU hits 100%. The most honest signal I’ve found isn’t origin 5xx rate - it’s edge retry rate divided by origin request rate. When that ratio starts climbing above 0.2, you’ve got amplification underway. Another early warning is the gap between edge p50 latency and origin p95 latency. If edge latency is flat but origin latency is climbing, your timeouts are about to start firing and retries are next. I keep a dashboard that plots these three together: retries per second, origin concurrency, and stale-serve rate. When retries climb and stale-serve stays flat, it means we’re not shedding load correctly.

The other metric teams ignore is origin connection queue time. Most origins will queue requests when saturated, and that queue time shows up before CPU does. If your load balancer or Nginx reports that time-to-first-byte from origin is jumping but origin CPU looks “okay,” your retries are already filling the queue. You want to alert on that queue time and automatically tighten the circuit breaker threshold when it exceeds your hedged request delay. Don’t wait for 5xx - by then the storm is fully formed. We fire a warning if retry attempts exceed 15% of origin traffic for more than 30 seconds, and we page if it hits 30%. Those thresholds sound low, but at 100k RPS, 30% retries is 30k extra requests that weren’t there a minute ago.

We baseline every deployment against a steady-state profile and compare during incidents. The table below is from a real incident we replayed in load testing - same origin pool, same edge logic, before and after adding jittered backoff, a retry budget, and per-PoP circuit breakers. The numbers aren’t theoretical; they’re what you see when you stop the edge from DDoSing your own origin. What you’re looking for is not just lower failures, but lower retry amplification and stable latency even when the origin is unhappy.

Putting Retry Controls Into Production: Budgets, Jitter, and Decorrelated Backoff

It's one thing to know you need a retry budget, it's another to actually enforce it at 180+ PoPs without shooting yourself in the foot. The most reliable setup we've run in production isn't a global counter — it's a per-PoP, per-origin token bucket that refills every second. We typically set the retry budget at 5% of successful requests over a 10-second rolling window, with a floor of 10 retries per second. That way a low-traffic PoP can still retry, but a high-traffic PoP in Ashburn doing 50k req/s doesn't suddenly unleash 15k retries when your origin blips for 400ms.

You've also got to fix your backoff. We see teams use a fixed 1-second retry and wonder why they get synchronized waves. Don't do it. You need decorrelated jitter with exponential base. For origin fetches, we don't retry faster than 50ms and we cap at 2 seconds. Each retry gets a new deadline: sleep = min(cap, rand_between(base, prev_sleep * 3)). That randomization alone dropped our 95th percentile origin concurrency during incidents by about 60%. And please, set aggressive per-try timeouts. If your origin normally responds in 80ms, a 750ms try timeout with a single retry is far safer than a 3-second timeout with three retries.

This is also where automation can bite you if you're not careful. An auto-scaler that sees retry-driven latency and spins up more edge workers just adds more retry sources. That's the core of The CDN Automation Reliability Paradox: When Automation Causes More Outages — your remediation amplifies the pressure. Wire your retry budget to your health checks, not your autoscaler.

# VCL / Edge Worker - Retry budget with decorrelated jitter
sub vcl_backend_fetch {
    # Token bucket: 5% of successes in last 10s, floor 10/s
    if (retry_budget_available(bereq.backend) == false) {
        return (abandon); # serve stale or 503, don't retry
    }

    set bereq.connect_timeout = 400ms;
    set bereq.first_byte_timeout = 750ms;
    set bereq.between_bytes_timeout = 500ms;
}

sub vcl_backend_error {
    if (beresp.status >= 500 && bereq.retries < 1) {
        # Decorrelated jitter: 50ms - 2000ms
        set beresp.retry_delay = random(50, min(2000, bereq.prev_delay * 3));
        # Consume 1 token from budget
        call consume_retry_token(bereq.backend);
        return (retry);
    }
    # Fallback to stale-while-revalidate if we have it
    if (stale.exists) { return (deliver_stale); }
}

Advanced Patterns: Coalescing, Shielding, and Debugging a Storm That Won't Quit

Once budgets and jitter are in place, the next biggest win is request coalescing — also called collapsing or single-flight. Without it, 8,000 clients asking for the same uncacheable or just-expired object during a 2-second origin slowdown becomes 8,000 origin fetches, plus retries. With coalescing enabled at the edge, those 8,000 collapse into one in-flight fetch and 7,999 clients wait for that single response. We've measured 40:1 to 100:1 reduction in origin load during cold-start events just from this. Pair it with an origin shield — a single tier-1 PoP per region that coalesces again — and you turn a global thundering herd into a handful of shield-to-origin requests.

If you're still seeing retry pressure after that, you're likely dealing with a downstream amplification loop. Here's how we troubleshoot it: First, compare edge_requests vs origin_requests vs origin_retries per PoP. If retries exceed 7-8% of origin requests for more than 30 seconds, your budget isn't enforcing. Second, check for hedging gone wrong — we saw one config where hedging after 120ms was enabled *and* retries were enabled, so every slow request generated 2x hedged requests plus a retry. Disable hedging for non-idempotent routes or when p50 origin latency already exceeds 200ms. Finally, look at stale serving. If your stale-while-revalidate hit rate drops below 85% during an incident, clients are waiting on fetches instead of getting stale content, which invites more retries.

Don't try to debug this live without a playbook. When your origin is flapping, you don't want to be grepping logs to guess which retry policy is active. This is exactly what CDN Runbook Automation: Converting Manual Procedures into Executable Scripts is for — codify the steps: disable retries globally via feature flag, force serve-stale for 5 minutes, drain shield queues, then re-enable budgets at 2% to test recovery. And after you survive it, do a proper review. We've found most retry storms have a second root cause hidden behind the first, which is why CDN Postmortem Analysis: Learning from Incidents to Prevent Recurrence matters — you'll want to chart retry count, budget exhaustions, and origin concurrency on the same timeline to see the feedback loop clearly.

Frequently Asked Questions