Thundering Herd Mitigation at CDN: Request Coalescing and Jittered Retries

I've been paged at 2:47 AM for a thundering herd exactly twice in my career, and once is more than enough. The first time, a single 45-second video segment expired from cache across 200 edge nodes simultaneously during a live sports final — about 80,000 clients asked for it within the same second and we forwarded every single request to a two-node origin. It didn't recover on its own. That's the moment you truly understand that CDN caching isn't just about hit ratio, it's about shielding your origin from your own success.

Why One Cold Object Can Avalanche Your Entire Tier

At its core, the thundering herd — or cache stampede — is a coordination failure. When a popular object expires, is purged, or has never been cached, every edge that receives a request in that window thinks it's responsible for fetching it. Without coordination, you don't have 200 edges serving a cache, you have 200 origins hammering one. I've seen this turn a routine 2,000 RPS origin into a pile trying to serve 60,000 RPS for a single URL. Your load balancer doesn't save you, your autoscaler can't react fast enough, and TCP queues just grow until everything times out.

Early in my time running a video CDN, we relied on short TTLs to keep content "fresh." It felt safe until we realized we’d created a synchronized timer for disaster. Every node expired the same object at precisely the same second. Clients naturally retry on failure, and without any damping, those retries line up perfectly with the next wave of real traffic. As described in the HTTP Working Group Specifications, cache validation and freshness directly govern this behavior, but the spec won't protect you from implementing it naively. The fix isn't longer TTLs alone — it's breaking that synchronization at the edge.

This is where reliability engineering has to meet content delivery head-on. You can’t treat herd mitigation as a one-off Nginx tweak; it’s a system property that touches capacity planning, monitoring, and how you define availability. We've folded our coalescing rules into our broader SRE Practices for CDN: Applying Site Reliability Engineering to Content Delivery runbooks because when it fails, it looks exactly like an origin outage, even though the origin was fine until we killed it.

# nginx: coalesce concurrent misses for the same key
# only the first request fetches from origin, others wait

proxy_cache my_zone;
proxy_cache_key $scheme$proxy_host$uri$is_args$args;
proxy_cache_lock on;              # enable request coalescing
proxy_cache_lock_timeout 5s;      # wait up to 5s for the leader
proxy_cache_lock_age 3s;          # if leader hasn't responded in 3s, allow another
proxy_cache_use_stale updating error timeout http_500 http_502 http_503;
proxy_cache_background_update on; # serve stale while revalidating in background

# Don't let waiting requests pile up forever
proxy_cache_lock_timeout 5s;
add_header X-Cache-Lock $upstream_cache_status;

Implementing Request Coalescing Without Starving Clients

Request coalescing — sometimes called collapsed forwarding or request collapsing — is simple in principle: when N requests arrive for the same uncached key, you let one through to the origin and park the other N-1. When the response comes back, you fan it out to everyone waiting. Varnish calls this grace, Cloudflare and Fastly do it internally, and even ATS has it. The hard part I’ve learned isn't turning it on, it's tuning the waiting.

If you set your lock timeout too short, you defeat the purpose and duplicate origin fetches. Too long, and you’re holding client connections open for seconds, burning worker threads and pushing p99 latency through the roof. We once set a 10-second lock timeout on our API edge and watched our connection pool saturate during a slow origin deploy — coalescing actually made the outage feel worse because clients weren’t failing fast. We also had to think about request methods and headers: you should only coalesce cacheable GET/HEAD requests with identical cache keys, including Vary headers. Coalescing a request with Authorization or a Range header onto the same origin fetch is how you serve private data to the wrong user.

For dynamic edges and workers, you can build this yourself with a shared promise map. You also need a circuit breaker so a hung origin fetch doesn't pin thousands of waiters forever. This pattern ties directly into how you spend your reliability budget — if your coalescing strategy adds 400ms to p95 but prevents 5 minutes of total origin collapse, that's a trade your CDN Error Budget Management: Balancing Reliability with Feature Velocity should explicitly account for. I’d rather burn a little latency budget than an entire error budget on a self-inflicted DDoS.

// Cloudflare Workers / edge worker: simple in-memory coalescing + stale-while-revalidate
const inflight = new Map();

async function fetchWithCoalescing(request, ctx) {
  const cacheKey = new Request(request.url, { method: 'GET' });
  const cache = caches.default;

  let cached = await cache.match(cacheKey);
  if (cached) return cached;

  // If another request is already fetching this URL, wait for it
  if (inflight.has(cacheKey.url)) {
    return inflight.get(cacheKey.url);
  }

  const originPromise = (async () => {
    try {
      const res = await fetch(request, { cf: { cacheTtl: 60 } });
      // Only cache successful, cacheable responses
      if (res.ok && res.headers.get('cache-control') !== 'no-store') {
        ctx.waitUntil(cache.put(cacheKey, res.clone()));
      }
      return res;
    } finally {
      inflight.delete(cacheKey.url);
    }
  })();

  inflight.set(cacheKey.url, originPromise);
  
  // Timeout guard: don't wait forever
  const timeout = new Promise((_, reject) => 
    setTimeout(() => reject(new Error('coalesce-timeout')), 3000)
  );
  return Promise.race([originPromise, timeout]);
}

function jitteredBackoff(attempt, baseMs = 200, capMs = 5000) {
  // Full jitter: random(0, min(cap, base * 2^attempt))
  const exp = Math.min(capMs, baseMs * Math.pow(2, attempt));
  return Math.random() * exp;
}

Calibrating Jitter So Retries Don't Resynchronize

Coalescing handles the first wave, but retries handle every wave after. The classic mistake I still see in client SDKs is a fixed 1-second retry: origin returns 503, all 10,000 clients wait exactly 1,000ms, then hit you again at the exact same millisecond. You've just rebuilt the herd you tried to dissolve. Even a simple exponential backoff without jitter still synchronizes because clients that failed together share the same start time.

What actually works is jittered exponential backoff, and the details matter. According to guidance on resilience patterns from web.dev Articles, you want either full jitter or decorrelated jitter, not just adding +/- 10% randomness. Full jitter — picking a random value between 0 and exp_backoff — spreads the retry thunder over the entire window. At the CDN edge itself, when your edge retries origin, you must also jitter. We use decorrelated jitter on our origin shield layer: next_sleep = random(base, prev_sleep * 3). It keeps median retries low but guarantees a long tail that prevents clustering. And we never retry non-idempotent methods or on 4xx, only on network errors, 429, 502/503/504.

We learned to automate the tuning too, because manually setting retry caps doesn't scale across 50+ services. Our synthetic monitors now measure origin retry amplification and we alert if edge retry RPS exceeds 10% of origin throughput — that's a sign your jitter isn't wide enough. That visibility came from the same push we did for CDN Toil Reduction Through Automation: Eliminating Repetitive Operations Work; once we stopped hand-editing retry configs per service, we could finally run experiments that proved our defaults were wrong.

Putting It Together: Production-Ready Coalescing with Jittered Origin Fetches

Theory's clean, but production's messy. At the edge, you'll typically implement coalescing at two layers: inside your cache (Varnish, ATS, Nginx) and at your origin shield. We learned the hard way that doing it at only one layer still leaves you exposed. If 200 PoPs all miss at once and each coalesces locally, you're still sending 200 synchronized origin requests. You need global coalescing at the shield tier.

For Nginx/OpenResty, proxy_cache_lock is your fastest win. Don't just turn it on — tune it. The defaults will absolutely starve clients. We run with proxy_cache_lock_timeout 8s and proxy_cache_lock_age 5s for most cacheable objects, which means waiters will queue for up to 8 seconds for the first fetch to complete, but if that fetch takes longer than 5 seconds we'll let a second request through to avoid a complete stall. For large objects (>5MB) we drop the timeout to 3s — you don't want 5,000 clients holding connections for 8 seconds eating up your worker conns. Pair this with jitter on the retry path and you've cut our origin thundering herd traffic by 92% during cold purges.

Here's the pattern we deploy on our shield Nginx tier that combines coalescing, stale-while-revalidate, and decorrelated jitter. The key is you don't retry immediately on failure — you sleep with jitter before the next coalesced leader retries:

# nginx shield tier - coalescing + jittered retries
proxy_cache_lock on;
proxy_cache_lock_timeout 8s;
proxy_cache_lock_age 5s;
proxy_cache_use_stale error timeout updating http_500 http_502 http_503;
proxy_cache_background_update on;
proxy_cache_valid 200 10m;
proxy_cache_valid 404 30s;

# jittered retry for origin failures - decorrelated jitter
# retry interval = min(cap, random_between(base, prev_interval * 3))
upstream origin {
    server origin.example.com max_fails=3 fail_timeout=30s;
}

proxy_next_upstream error timeout http_502 http_503;
proxy_next_upstream_timeout 15s;
proxy_next_upstream_tries 3;

# In OpenResty/Lua for finer control:
# local base = 100  -- ms
# local cap = 5000  -- ms
# local prev = 100
# for i=1,3 do
#   local sleep = math.random(base, math.min(cap, prev * 3))
#   ngx.sleep(sleep/1000)
#   prev = sleep
# end

Advanced Patterns: Grace Mode, Probabilistic Expiration, and Debugging Stuck Locks

Once basic coalescing is stable, you'll hit the next class of problems: stuck locks and synchronized expirations. A single slow origin response (we've seen 45s+ TTFB from a struggling app server) can hold a coalescing lock and queue 10k+ waiting clients. That's not mitigation, that's a self-inflicted DDoS. You need proxy_cache_background_update and grace/stale-while-revalidate. Serve stale for 60-120 seconds while a single background thread revalidates. Varnish's beresp.grace or Fastly's stale-while-revalidate does this elegantly. We serve stale for 90s with a background fetch — p95 latency stays under 40ms even when origin p99 spikes to 2.8s. Without it, that same origin blip pushes edge p95 to 4.2s.

The other killer is synchronized TTLs. If you set Cache-Control: max-age=3600 on a hot object, every edge node expires it at the same second and you get a periodic herd every hour. Fix it with probabilistic early expiration (also called jittered TTL or x-fetch randomization). Instead of expiring exactly at 3600s, expire randomly between 3300s and 3900s. In Varnish: set beresp.ttl = 3600s + std.random(0, 600) - std.random(0, 600);. For high-cardinality workloads, we also use the The CDN Automation Reliability Paradox: When Automation Causes More Outages as a caution — we once automated a global purge that reset TTLs everywhere identically. Randomization would have prevented that synchronized miss storm.

When it breaks, here's how to debug it. First, export metrics: cache_lock_waiters, cache_lock_timeouts, and origin_concurrency_per_object. If waiters > 500 for a single key for more than 10s, your lock is too long. If you see lock timeouts spiking >2% of cache misses, your origin is too slow or your lock_age is too short. Second, don't automate recovery blindly. We've codified this in our CDN Runbook Automation: Converting Manual Procedures into Executable Scripts — any auto-unlock script must check that origin health is OK first, otherwise you're just re-triggering the herd. And after every incident, run a proper CDN Postmortem Analysis: Learning from Incidents to Prevent Recurrence to answer: was it a true herd or a retry storm that re-synchronized? The fix is different.

Frequently Asked Questions