Cache Stampede Prevention: Request Coalescing and Locking

A cache stampede occurs when a cached object expires and many simultaneous requests bypass the empty cache to hit the origin server at once. This phenomenon, also known as the thundering herd problem, can overwhelm origin infrastructure and cause cascading failures across your delivery pipeline. Preventing stampedes is essential for maintaining the origin protection that CDNs are designed to provide, and every operations team running high-traffic systems must understand the available mitigation strategies.

What Is the Thundering Herd Problem?

The thundering herd problem describes a scenario where a popular cached resource expires, and every concurrent request for that resource simultaneously discovers the cache miss. Instead of one request reaching the origin, hundreds or thousands arrive within milliseconds. The origin server, sized to handle the trickle of cache-miss traffic under normal conditions, suddenly faces a load spike orders of magnitude above its capacity. For foundational context, read CDN Caching Strategies: Pull, Push, and Hybrid — FlyFish NWA.

Consider a product page cached with a 60-second TTL that receives 5,000 requests per second. Under normal operation, the CDN serves all 5,000 requests from cache, and the origin handles perhaps one request per minute for revalidation. When the TTL expires, all 5,000 requests in that second discover the cache is empty. If the origin takes 200 milliseconds to respond, another 1,000 requests arrive before the cache is refilled. The origin now faces 6,000 concurrent requests instead of the usual one.

This problem intensifies with multi-tier cache hierarchies. When an L1 edge cache expires, it forwards to L2. If L2 has also expired, the stampede reaches the origin with amplified force because multiple L1 nodes simultaneously send requests through L2.

How Does Request Coalescing Work?

Request coalescing, also called request collapsing, is the primary defense against cache stampedes. When multiple requests arrive for the same cache key during a cache miss, the CDN forwards only the first request to the origin and holds all subsequent requests at the edge. Once the origin responds, the CDN populates the cache and serves the response to all waiting clients simultaneously.

The mechanism works as follows:

1. Request A arrives, cache miss detected, forwarded to origin.
2. Requests B through Z arrive for the same URL while A is in flight.
3. CDN recognizes the pending origin fetch and queues B through Z.
4. Origin responds to A, CDN caches the response.
5. CDN serves the cached response to B through Z immediately.

This reduces origin load from N concurrent requests to exactly one, regardless of traffic volume. The tradeoff is that coalesced requests experience additional latency equal to the remaining origin response time when they arrive.

Coalescing in Nginx

Nginx implements request coalescing through its proxy cache lock directive:

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=cdn:100m;

location / {
    proxy_cache cdn;
    proxy_cache_lock on;
    proxy_cache_lock_timeout 10s;
    proxy_cache_lock_age 5s;
    proxy_pass http://origin;
}

The proxy_cache_lock on directive enables coalescing. The proxy_cache_lock_timeout sets how long a request will wait before bypassing the lock and making its own origin fetch. The proxy_cache_lock_age controls how long after the first request was sent before the lock is released, allowing another request to try if the first has not returned.

Coalescing in Varnish

Varnish Cache uses a mechanism called request coalescing by default. When the first request for an expired object triggers a backend fetch, subsequent requests for the same object wait for that fetch to complete. The behavior is controlled in VCL:

sub vcl_backend_response {
    # Set grace period for stale serving during coalescing
    set beresp.grace = 30s;

    # Set keep period for conditional requests
    set beresp.keep = 5m;
}

Varnish's grace mode allows it to serve the stale cached version to waiting requests while the backend fetch completes, combining coalescing with stale serving for optimal performance.

What Is Cache Locking and When Should You Use It?

Cache locking is a more granular form of stampede prevention. When a cache entry expires, the first request acquires a lock on that cache key. All subsequent requests for the same key see the lock and either wait for the lock to release or receive a stale version. Only the lock holder fetches from the origin.

Cache locking differs from basic coalescing in several ways:

• Explicit lock state: The cache maintains a visible lock per key, not just an implicit pending-fetch state.
• Lock timeout: If the lock holder fails or the origin is slow, the lock expires and another request takes over.
• Stale fallback: Locked-out requests can serve stale content rather than waiting, eliminating the latency penalty of coalescing.
• Distributed locks: In multi-node deployments, distributed cache locks prevent multiple edge nodes from simultaneously fetching the same resource through the cache hierarchy.

Use cache locking when your content requires strict freshness guarantees and you cannot afford to serve stale data. Financial data dashboards, real-time inventory systems, and live sports scores are common use cases.

How Does Probabilistic Early Revalidation Prevent Stampedes?

Probabilistic early revalidation takes a fundamentally different approach. Instead of handling the stampede after it occurs, it prevents the stampede from forming by refreshing cached content before the TTL expires.

Each incoming request calculates a probability of triggering a background revalidation. The probability starts near zero when the content is freshly cached and increases as the TTL approaches expiration. The formula typically follows:

probability = (elapsed_time / ttl) ^ exponent

# Example with 300s TTL, exponent=3:
# At 200s elapsed: (200/300)^3 = 0.30 (30% chance)
# At 250s elapsed: (250/300)^3 = 0.58 (58% chance)
# At 290s elapsed: (290/300)^3 = 0.90 (90% chance)

For a resource receiving 100 requests per second, the cache will almost certainly be refreshed well before expiration. The beauty of this approach is that it scales naturally with traffic: high-traffic resources get refreshed earlier because more requests roll the probabilistic dice, while low-traffic resources that would not cause stampedes anyway refresh closer to expiration.

This technique is sometimes called XFetch or probabilistic early expiration in academic literature. It requires no coordination between edge nodes and adds negligible computational overhead per request.

How Can Stale-While-Revalidate Mitigate Stampedes?

The stale-while-revalidate (SWR) directive provides a built-in HTTP mechanism for stampede mitigation. When a cached response exceeds its max-age but remains within the stale-while-revalidate window, the CDN serves the stale response immediately while triggering a background revalidation:

Cache-Control: max-age=300, stale-while-revalidate=60

This header tells the CDN to serve cached content for 300 seconds. Between 300 and 360 seconds, serve the stale version instantly while fetching a fresh copy in the background. After 360 seconds, treat it as a true cache miss.

SWR is effective against stampedes because the stale response is served immediately. There is no waiting, no coalescing delay, and no lock contention. The background revalidation is inherently serialized because only one request triggers it, achieving the same origin-protection effect as coalescing but with zero additional user-facing latency. For a deeper understanding of TTL optimization, consider how SWR windows complement your overall TTL strategy.

How Do Prevention Strategies Compare?

Each stampede prevention strategy has distinct characteristics suited to different scenarios. The following table compares them across key dimensions:

In production, these strategies are rarely used in isolation. A robust stampede prevention architecture combines SWR for routine expirations, request coalescing as a safety net for cold-cache scenarios, and proactive cache warming for anticipated traffic surges.

How Do You Protect the Origin During an Active Stampede?

When stampede prevention mechanisms fail or a cold-cache event occurs, you need emergency origin protection:

Connection Queuing

Limit the number of concurrent connections from CDN to origin. Excess requests queue at the edge rather than overwhelming the origin. Configure per-server connection limits at the load balancer level:

# Nginx upstream with connection limiting
upstream origin {
    server origin1.example.com max_conns=50;
    server origin2.example.com max_conns=50;
    queue 1000 timeout=30s;
}

Circuit Breaker Pattern

When origin error rates exceed a threshold, the circuit breaker opens and the CDN stops sending requests. During this period, the CDN serves stale content, custom error pages, or reduced-functionality responses. After a cooldown period, the circuit half-opens, allowing a limited number of test requests through. If they succeed, the circuit closes and normal operation resumes.

Adaptive Retry Backoff

If origin requests fail, implement exponential backoff with jitter to prevent retry storms from compounding the stampede. Random jitter ensures that retries from different edge nodes do not synchronize and create secondary stampedes.

What Are Real-World Implementation Patterns?

Large-scale CDN deployments use layered stampede prevention:

Layer 1 -- SWR headers: Set stale-while-revalidate on all cacheable responses. This handles 95% of potential stampedes with zero configuration at the CDN layer. Work with your Cache-Control header strategy to set appropriate windows.

Layer 2 -- Request coalescing: Enable at both L1 edge and L2 mid-tier caches. This catches cold-cache scenarios where no stale version exists, which SWR cannot address.

Layer 3 -- Origin shield: Funnel all cache misses through a single origin shield node that performs its own coalescing. This collapses stampedes from hundreds of edge nodes into a single origin request.

Layer 4 -- Warming: For product launches, marketing campaigns, and other anticipated traffic events, use cache warming scripts to pre-populate edge caches before the traffic arrives.

How Do You Monitor for Stampede Events?

Detecting stampedes requires monitoring several metrics in combination:

• Cache hit ratio drops: A sudden decrease in cache hit ratio indicates mass cache misses, the hallmark of a stampede.
• Origin request spike: A sharp increase in origin requests per second correlated with cache expiration timing.
• Response time increase: Origin response time climbs as it becomes overloaded, further exacerbating the stampede duration.
• Coalescing queue depth: The number of requests waiting behind a coalescing lock. High values indicate effective coalescing under pressure.
• 5xx error rates: Origin errors during stampede events signal that protection mechanisms were insufficient.

Set alerts on the ratio of origin requests to total requests. Under normal operation, this ratio should be well below 1%. A spike above 5% warrants investigation. Correlate these alerts with TTL schedules to identify whether the stampede was caused by synchronized cache expiration.

Understanding how the full CDN request flow operates from browser to edge to origin helps identify the exact point where stampede amplification occurs and apply the right countermeasure at the right layer.

Cache stampede prevention is not a single configuration toggle. It is a layered defense strategy that combines HTTP-level directives, CDN-platform features, and origin-side protections. The most resilient architectures treat every cache expiration as a potential stampede and design multiple fallback mechanisms to ensure the origin never receives more than a controlled trickle of traffic, even under the most demanding conditions.