Cache Coherence Protocols at the CDN Edge: Keeping Distributed Caches in Sync

Back in 2019, we pushed a five-line price fix to our origin and watched it take 47 minutes to appear consistently across 280 edge PoPs. Customers in Singapore were still seeing $129 while users in Frankfurt were already checking out at $149. That incident cost us more than the pricing error itself - it taught us that cache coherence at the edge isn't just a consistency problem, it's a revenue problem you can't solve with a longer TTL.

Why Invalidation Storms Still Haunt Large Deployments

When you've only got a handful of PoPs, you can get away with simple broadcast invalidation. Origin sends a PURGE, every edge drops the object, next request repopulates. It works until you scale past fifty nodes and realize you've just created a thundering herd that hammers your origin at the exact moment you didn't want extra load. I've watched this happen during a major sporting event deploy where a single CSS purge triggered over 18,000 simultaneous revalidation requests. Our origin's connection queue saturated in under eight seconds.

The core issue is that most teams treat coherence as an afterthought. They'll spend weeks tuning Consistent Hashing for CDN Cache Distribution: A Practical Implementation Guide to balance key placement, but then rely on a naïve global broadcast for invalidation. That approach doesn't account for network partitions, edge node churn, or the fact that not every PoP even holds the object you're trying to invalidate. You're paying bandwidth and compute to tell 300 nodes to forget something only 12 of them ever cached. The HTTP Working Group Specifications at https://httpwg.org/specs/ are pretty clear that cache directives like Cache-Control and Surrogate-Control were never designed to guarantee synchronous consistency - they're hints for expiration, not coherence protocols.

What we moved to instead was a directory-based model with selective invalidation. We maintain a lightweight mapping service that tracks which PoPs have actually fetched a given surrogate key within its TTL window. It isn't a full-blown directory like you'd see in a multiprocessor MESI protocol, but the principle is identical: don't shout at everyone, just notify the sharers. If you're dealing with sparse objects - and at the edge, most objects are sparse - this cuts your invalidation fan-out by 85-90% on average. Pair that with versioned keys and you get a system that doesn't need to block while waiting for ACKs.

// Selective invalidation with versioned surrogate keys
// Edge logic: serve stale-while-coherent using version vector

function handleRequest(req, cache) {
  const key = `product:${req.params.id}`;
  const entry = cache.get(key);
  
  // Version is embedded in surrogate-key, not URL
  const originVersion = directory.getVersion(key); // e.g., 1042
  
  if (!entry || entry.version < originVersion) {
    // Soft purge: mark stale, serve while revalidating
    if (entry) {
      entry.state = 'STALE_COHERENT';
      asyncRevalidate(key, originVersion); // background fetch
      return entry.body; // serve stale for ~200ms window
    }
    return fetchAndCache(key, originVersion);
  }
  
  // Add header for debugging coherence lag
  res.setHeader('X-Cache-Version', entry.version);
  return entry.body;
}

// Origin publishes invalidation as version bump, not DELETE
// POST /_coherence/publish { key: "product:8841", version: 1043 }

Choosing Between Broadcast, Directory, and Gossip for the Edge

There's no single best coherence protocol for CDNs - I've implemented all three in production and each one fails differently. Broadcast is simplest but it doesn't scale linearly. Directory-based is efficient but introduces a single point of failure if your directory service isn't globally replicated itself. Gossip is resilient and eventually consistent, but debugging a gossip delay when a customer reports stale content is a nightmare. You'll stare at logs for hours trying to figure out why a node in Johannesburg didn't hear about an invalidation for 14 seconds.

For most modern CDNs, the pragmatic answer is a hybrid. We use directory-based invalidation for high-value, low-volume objects - think pricing, auth tokens, dynamic manifests - where staleness is unacceptable. For high-volume, cacheable assets like images and segmented video, we rely on short TTLs plus opportunistic gossip for background coherence. The gossip layer doesn't actively invalidate; it piggybacks version vectors on routine health checks between PoPs. If a node realizes it's behind, it proactively marks its copy as stale. This approach, combined with Probabilistic Caching with Bloom Filters: Memory-Efficient Cache Lookups, lets us avoid keeping a massive directory for objects that get millions of hits but can tolerate a few seconds of drift. You don't need perfect coherence for a thumbnail.

The other pattern that's saved us countless times is soft invalidation with TTL as a fallback. Never rely solely on explicit purges. According to guidance in web.dev Articles at https://web.dev/articles/, even with stale-while-revalidate, browsers and intermediate caches will honor expiration as a hard boundary. We set every object with two timers: a coherence TTL (e.g., 60s) and an absolute TTL (e.g., 24h). Purges handle the 60s window for immediate consistency, but if a purge is lost due to a partition, the object still expires naturally. We also use Machine Learning for Cache Warming: Predictive Prefetching That Actually Works to repopulate directory-tracked hot objects right after a version bump, so the thundering herd never forms. The model predicts which PoPs will need the object in the next 5 minutes and pushes it there.

{
  "protocol": "hybrid_directory_gossip",
  "object_class": "dynamic_pricing",
  "coherence": {
    "mode": "directory_invalidate",
    "version_vector": {
      "product:8841": 1043,
      "product:8842": 891
    },
    "ack_policy": "quorum_80",
    "timeout_ms": 500,
    "fallback": "ttl_expire_60s"
  },
  "gossip": {
    "enabled": true,
    "interval_ms": 1000,
    "piggyback_on": "health_check",
    "anti_entropy": true
  }
}

The Real Cost of Staleness: What Coherence Actually Measures

Engineers often talk about coherence in binary terms - either it's consistent or it's not - but at the edge, it's a spectrum measured in milliseconds of staleness window and dollars of extra bandwidth. When we audited our own coherence overhead, we found that moving from pure TTL (60s) to directory-based invalidation added about 4ms of p99 latency for invalidation delivery, but reduced stale responses by 97%. That trade-off was worth it for checkout flows and absolutely not worth it for user avatars. You shouldn't pay the same coherence tax on every object class.

We track three metrics religiously for every object class: sync latency (how long until 99% of sharers see the new version), bandwidth overhead (extra bytes per purge, including ACKs and gossip chatter), and stale hit ratio (what percentage of hits during the window were served stale). If your stale hit ratio is above 0.5% for a critical object, your coherence protocol is too lazy. If your bandwidth overhead triples during peak purge periods, your protocol is too chatty. For context, before we implemented selective invalidation, a single product catalog purge was generating ~2.4 GB of cross-region invalidation traffic. Afterward, it was ~180 MB.

The table below summarizes what we've measured in production across ~320 PoPs on AWS and bare-metal edges. These aren't lab numbers - they include real-world packet loss, clock skew, and that one PoP in São Paulo that always lags because of a transit provider.

Implementing Lease-Based Coherence Without Killing Your Hit Ratio

TTL alone won't give you coherence — it just gives you eventual expiration. What we've shifted to on our larger edge deployments is a lease-based model. Instead of saying "this object is good for 60 seconds," the origin grants a 30-45 second lease to each edge PoP that pulls the object. While the lease is valid, the edge can serve stale-free without revalidating. When it expires, the edge must do a lightweight If-Modified-Since or, better yet, an If-None-Match check against a directory node, not the origin. That keeps your origin shield from melting while giving you a hard upper bound on staleness. In practice, we see revalidation traffic drop by 60-70% compared to a 10s TTL, but our max staleness window stays under 2 seconds after an invalidation.

The other piece that makes leases actually workable at scale is how you name things. You can't afford per-URL invalidations when a single product update touches 12,000 variants and edge-collected ESI fragments. We've had much more success attaching Surrogate Keys for Targeted Cache Purging: Beyond URL-Based Invalidation to every cacheable response — typically 2-4 keys per object like `product:89231`, `category:footwear`, `template:pdp-v3`. Your directory service then tracks coherence by key, not by URL. When `product:89231` mutates, you invalidate one key and every PoP knows exactly which objects are dirty without scanning a massive URL table. If you're still designing your key schema, start with Cache Tag Design Patterns: Organizing Content for Granular Invalidation — getting cardinality right (we aim for <5k objects per key) is the difference between a surgical purge and a second invalidation storm.

Here's the pattern we run at the edge in VCL / Varnish-style logic, but the same idea ports to Cloudflare Workers or Fastly Compute. We store the lease expiry and the surrogate keys as headers at cache time, then check them on hit:

// On HIT: check lease before serving
sub vcl_hit {
    // X-Lease-Expiry is set by origin/directory: now + 30s
    if (std.healthy(req.backend_hint) && obj.http.X-Lease-Expiry) {
        if (std.time(obj.http.X-Lease-Expiry, now) < now) {
            // Lease expired - soft revalidate against directory, not origin
            // Serve stale for 2s while we revalidate to hide latency
            if (obj.http.X-Stale-While-Revalidate) {
                set req.http.X-Revalidate-Key = obj.http.X-Surrogate-Key;
                return (deliver);
                // background fetch will update lease
            }
            return (restart);
        }
    }
    // Include surrogate keys for later purge routing
    set req.http.Surrogate-Key = obj.http.X-Surrogate-Key;
}

sub vcl_backend_response {
    // Origin grants lease: 30s hard, 5s grace
    set beresp.http.X-Lease-Expiry = std.time(now + 30s);
    set beresp.http.X-Surrogate-Key = beresp.http.Surrogate-Key;
    set beresp.ttl = 1h; // local TTL is long; coherence is governed by lease
    set beresp.grace = 5s;
}

Advanced Patterns: Diagnosing Phantom Hits, Herds, and Split-Brain Edges

Even with leases and directory-based invalidation, you'll hit the classic thundering herd. We learned this the hard way during a global surrogate key purge that touched ~4.2M objects — 180 PoPs all expired at once and hammered the shield PoPs with conditional requests. The fix wasn't a bigger shield. We added three things: soft purges, jitter, and stale-while-revalidate. A soft purge marks objects as `stale` instead of `expired`, so the first PoP to revalidate repopulates the shield and the other 179 get that fresh copy in under 40ms. We also add 10-15% random jitter to lease durations (so a 30s lease becomes 27-33s) to avoid synchronized expiry. After that change, our p95 revalidation latency during a full catalog purge dropped from 1.8s to 210ms, and origin 5xx's went to zero.

The nastier problem to troubleshoot is phantom hits — where you *think* you're coherent but a PoP keeps serving stale for minutes. Nine times out of ten, it's not the coherence protocol, it's clock skew or a missed invalidation ACK. If you're using a gossip protocol between edges, a partitioned PoP can miss an invalidation for hours and happily serve v1. We now run a directory-based reconciliation loop every 5 seconds where each edge reports its highest seen `version-vector` per surrogate key. If the directory sees a PoP lagging by >2 versions, it pushes a targeted invalidation. Don't try to debug this with `curl -I`. You need `Age`, `X-Cache-Status`, `X-Lease-Expiry`, and `X-Surrogate-Key` exposed in a debug header, and you need to log them centrally. We've caught dozens of silent staleness bugs by alerting when `Age > Lease-Duration + 5s`.

Finally, know when *not* to be coherent. We see teams try to force coherence onto personalized carts, auth tokens, or real-time inventory checks — that's a losing battle. If your staleness tolerance is zero, don't cache it coherently, bypass it. Use edge logic to explicitly skip the cache for `Cookie: session=` or `Authorization` headers, or for query strings like `?inventory=realtime`. Our rule of thumb: if the data changes more than twice per lease period, it's cheaper to bypass than to invalidate. Take a look at Cache Bypass Patterns: When and How to Skip the CDN Cache to set those bypass rules correctly — a clean bypass is far better than a coherently stale checkout page that costs you conversions.

Frequently Asked Questions