Surrogate Keys for Targeted Cache Purging: Beyond URL-Based Invalidation

I got paged at 2:13 a.m. because a price change for a single SKU wasn't showing up anywhere. The retailer had updated one product, but that product appeared on 14,000 cached URLs — category pages, search results, recommendation carousels, and localized variants. We'd fired off a URL purge script and it was still churning through the queue 20 minutes later, while customers were checking out with stale prices. That's the moment I stopped trusting URL-based invalidation for anything dynamic.

Why URL-Based Purging Falls Apart at Scale

URL purging feels logical at first. You cache a response for /products/12345, and when that product changes, you send a purge request for that exact URL. Most CDNs make this dead simple — a single API call with a path, and the edge drops that object. For a small blog or a mostly static marketing site, it works fine and we've run it that way for years without issues.

The problem isn’t the mechanism, it’s the math. Modern pages aren’t one-to-one with data. That same SKU 12345 might be rendered on its product detail page, on three category listings, on a “new arrivals” shelf, on a personalized “you might also like” block, and on 12 different cached variants for currency and language. When the merchandising team updates the product title or inventory status, you don’t have one URL to purge — you have thousands. I’ve audited e-commerce sites where a single product update touched over 80,000 cached objects once you counted all the edge variants. Trying to track and purge them by URL turns into a distributed bookkeeping nightmare. You’re either maintaining a reverse index in your application just to know what to purge, or you’re firing off wildcard purges that nuke your hit ratio.

And wildcard or prefix purges aren’t a free lunch either. Purge by prefix like /category/shoes* is fast, but it’s a blunt instrument. You’ll evict thousands of objects that didn’t need to be invalidated just to get the ten that did. That thundering herd back to origin right after a big purge is often worse than the stale content you were trying to fix. This is where understanding Cache Coherence Protocols at the CDN Edge: Keeping Distributed Caches in Sync helps, because it shows why invalidation needs to be precise rather than broad, especially when you’re trying to keep hundreds of PoPs consistent without crushing origin.

# The old way: purging by URL (and why it doesn't scale)
# This script had to run for every product update

curl -X POST "https://api.cdn-provider.net/purge" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": [
      "/p/12345",
      "/category/shoes?page=1",
      "/category/shoes?page=2",
      "/api/recommendations?sku=12345",
      "/fr/p/12345",
      "/de/p/12345"
      // ... and 13,994 more we had to look up in the database
    ]
  }'

# Result: 429 rate-limited, partial purge, stale content still serving
# p95 purge propagation: 4-7 minutes for full URL list

Tagging Responses with Surrogate Keys: Grouping by Meaning, Not Location

Surrogate keys — sometimes called cache tags or cache keys depending on the vendor — flip the whole model. Instead of saying “purge this URL,” you tag each cached response with one or more logical identifiers when it’s stored, and then you purge by that identifier. Think of them as labels you stick on the cached object at the origin. Fastly calls them Surrogate-Key, Cloudflare and Akamai have Cache-Tag, but the idea is identical.

When your origin responds, it adds a header like Surrogate-Key: product-12345 category-shoes brand-nike. The CDN strips that header before delivering to the client (users never see it) but stores the mapping internally. That product page, the category page, and the API response for recommendations all get tagged with product-12345 if they include that product. Later, when product 12345 changes, you send one purge request: purge key product-12345. The CDN then instantly invalidates every object that was stored with that key, no matter what its URL was. You don’t need to know the URLs anymore, you just need to know what changed.

I first implemented this for a media customer with a headless CMS. An author would update a single article, and that article was embedded on the homepage, three section fronts, a dozen tag pages, and an RSS feed. With URL purging we were doing 20+ API calls per edit and inevitably missing a few. With surrogate keys we tagged each response with article-9876 author-42 tag-politics. An edit became one API call that purged in under 150ms globally. If you’ve looked at Probabilistic Caching with Bloom Filters: Memory-Efficient Cache Lookups, the concept is similar — you’re trading a direct lookup for a much more efficient indirection. The spec behind this builds on standard HTTP caching semantics documented in the HTTP Working Group Specifications, and modern best practices for controlling that behavior are well-covered in web.dev Articles on cache control. For warming, we later paired it with Machine Learning for Cache Warming: Predictive Prefetching That Actually Works to prefetch only the tagged groups we’d just purged.

# Origin response with surrogate keys (Varnish / Nginx / App layer)
# Example: Node.js/Express middleware adding tags

app.get('/category/shoes', async (req, res) => {
  const products = await db.getProductsByCategory('shoes');
  const tags = products.map(p => `product-${p.id}`);
  tags.push('category-shoes', 'template-category-grid');

  // Up to ~100 keys per response is typical limit (varies by provider)
  res.set('Surrogate-Key', tags.join(' '));
  res.set('Cache-Control', 'public, max-age=86400');
  res.set('Surrogate-Control', 'max-age=86400');
  res.render('category', { products });
});

# Purging becomes trivial and atomic
curl -X POST "https://api.cdn-provider.net/purge/tag" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"surrogate_keys": ["product-12345"]}'

# Response: {"status": "purged", "keys": ["product-12345"], "purged_objects": 14203}

Designing a Key Schema That Won't Collapse Under Its Own Weight

Throwing tags at every response feels powerful until you create a new set of problems. I’ve seen teams generate surrogate keys dynamically from database IDs without any constraints and end up with 5,000 unique keys on a single homepage. Most CDNs enforce limits — usually 64 to 128 keys per object and a header size limit around 16KB — and if you exceed them, keys get truncated silently and purges miss objects. You won’t know until you’re debugging stale content at midnight.

A good schema is deliberately low-cardinality and hierarchical. We typically use three tiers: entity keys like product-12345 or article-9876, collection keys like category-shoes or author-42, and template or dependency keys like template-product-card or config-pricing-v2. That last tier is crucial. When you ship a frontend template change that affects how prices render, you can purge template-product-card and invalidate every page using that component without knowing which products are on them. Keep key names short, lowercase, hyphens only, and don’t encode user-specific data into them. If you’re tempted to create user-12345-cart, you probably shouldn’t be caching that response at the edge at all.

You also need to decide on soft vs. hard purging. A hard purge deletes the object immediately and the next request must go to origin. A soft purge (or stale-while-revalidate) marks it as expired but allows the CDN to serve stale content while it fetches a fresh copy in the background. For product inventory where serving a stale price for another 2 seconds is better than a cache miss storm, soft purging by surrogate key has saved our origins more than once. The tradeoff is always between consistency and origin load, and you should measure it instead of guessing.

Implementing Surrogate Keys in the Real World: Headers, VCL, and Purge Wiring

Alright, you've got a clean schema on the whiteboard — now you've got to wire it into the actual request path. The good news is most modern CDNs make this pretty painless, but the devil's in where you generate the keys. Don't do it at the edge if you can avoid it. Your origin knows the most about an object's dependencies, so have your application or origin server emit the Surrogate-Key (Fastly) or Cache-Tag (Cloudflare, Akamai) header directly. The CDN just needs to store and index it.

We typically set this in vcl_fetch or your origin's response headers and strip it before delivery to the client — there's no reason to leak internal tagging logic to browsers. Keep it lean: aim for 5 to 15 keys per object. Fastly caps you at 1,000 keys per object and a 16KB header limit, Cloudflare Enterprise limits you to around 1,000 tags per response as well, but you'll see purge latency degrade long before that. If you're tagging more than ~30 keys per object, your schema's probably too granular. And watch your character set — stick to alphanumeric, dashes, and underscores. A single bad character will cause the entire header to be ignored on some platforms.

# Origin response headers (what your app should emit)
Surrogate-Key: product-12345 collection-running-shoes brand-nike template-pdp
Cache-Control: public, max-age=86400, stale-while-revalidate=300
Surrogate-Control: max-age=86400

# Fastly VCL - store the keys and hide them from clients
sub vcl_fetch {
  # beresp.http.Surrogate-Key is already set by origin
  # You can also append edge-generated keys if needed:
  # set beresp.http.Surrogate-Key = beresp.http.Surrogate-Key + " pop-" + server.pop;
  
  set beresp.ttl = 86400s;
  set beresp.grace = 24h;
}

sub vcl_deliver {
  # Don't leak internal keys to the browser
  unset resp.http.Surrogate-Key;
  unset resp.http.Surrogate-Control;
}

# Purging by key - fanout is ~150-300ms globally
# curl -X POST "https://api.fastly.com/service/YOUR_SERVICE_ID/purge/product-12345" \
#   -H "Fastly-Key: YOUR_API_TOKEN" -H "Fastly-Soft-Purge: 1"

Beyond the Basics: Soft Purges, Key Explosions, and Stale Content Ghosts

Once you're past the initial implementation, you'll hit the fun edge cases. First up: hard vs. soft purges. A hard purge evicts the object instantly, so the next request pays the full origin fetch penalty. At scale, if you hard-purge a key like collection-shoes that touches 40,000 URLs, you'll thundering-herd your origin. We almost always default to soft purges — the CDN marks the content as stale but still serves it for 15-30 seconds while it revalidates in the background. Pair that with stale-while-revalidate and you can purge 50k objects without a spike in origin 5xx errors. It's not quite instant, but users don't notice a 200ms stale response nearly as much as they notice a 3-second cache miss storm.

The second killer is high-cardinality key explosion. I've seen teams tag every object with user-12345 or session-abc and then wonder why their hit ratio tanks and purge API calls start getting rate-limited. If a key only ever matches one or two objects, it's not a surrogate key — it's just a URL in disguise. Audit your keys weekly. If any key is associated with fewer than ~10 objects or you're generating more than 10,000 unique keys per day, you've got a design problem. That's where a solid Cache Tag Design Patterns: Organizing Content for Granular Invalidation review pays for itself, and knowing when not to tag is just as important — check your Cache Bypass Patterns: When and How to Skip the CDN Cache for personalized or uncacheable content. Don't cache it just to purge it later.

Finally, don't forget that purging is a distributed systems problem. A surrogate key purge isn't a single database delete; it's a broadcast to every POP. Most providers use an eventually consistent model that relies on Consistent Hashing for CDN Cache Distribution: A Practical Implementation Guide under the hood. That means you'll see 100-250ms propagation on Fastly or Cloudflare, but up to 2-3 seconds on larger Akamai footprints. If you immediately re-request a URL after purging and still get stale content, you didn't do anything wrong — you just hit a POP that hadn't received the invalidation yet. Build a 1-2 second delay into your CMS publish webhook or, better yet, use surrogate keys to version your deploys instead of relying on synchronous purges.

Frequently Asked Questions