HTTP Caching Headers: Cache-Control, ETag, Vary and Conditional Requests

Every byte your origin doesn't have to send is a win for latency, cost, and reliability. Yet caching remains one of the most misunderstood parts of HTTP, even among experienced developers. I have spent years tuning edge configurations where a single misplaced directive turned a highly cacheable workload into a stream of origin hits, or where a missing Vary header caused users to see the wrong language or broken compressed responses. HTTP caching is not magic; it is a small set of headers — Cache-Control, ETag, Last-Modified, and Vary — and the conditional request mechanism that ties them together. Get them right, and your CDN does the heavy lifting. Get them wrong, and you are effectively bypassing the edge entirely.

Cache-Control Directives: The Single Source of Truth for Freshness

If you configure only one header, make it Cache-Control. Introduced in HTTP/1.1, it replaced the older Expires header and gives you precise, composable control over how long a response is considered fresh and who is allowed to store it. In my experience, most caching bugs come from either omitting Cache-Control entirely or copying a snippet from Stack Overflow without understanding what each token does.

A response is fresh as long as its age is less than the freshness lifetime. During that window, a browser or CDN can serve it immediately without contacting the origin. Once it becomes stale, it must be revalidated or discarded depending on other directives.

Response Directives You Will Actually Use at the Edge

For CDN and browser caching, these are the directives that matter day-to-day:

public means any cache may store the response, even if the request was authenticated. Use this for static assets. private restricts storage to the end-user's browser only — the CDN must not cache it. I have seen APIs accidentally send private on public, cacheable JSON, which completely defeats edge caching. max-age=<seconds> sets the freshness lifetime relative to the time the response was generated. s-maxage overrides max-age but only for shared caches like CDNs. This lets you keep a short browser TTL while allowing a longer edge TTL.

no-store is the hard opt-out. It forbids storing the response anywhere. Use it for sensitive data like banking pages or personal information. no-cache is confusingly named — it does not prevent storage. It forces revalidation on every use. The cache can store it, but it must check with the origin before serving it. must-revalidate says that once stale, the cache must not serve the stale copy even if the origin is unreachable. Without it, some caches may serve stale on error.

Two newer directives have become essential for edge delivery: stale-while-revalidate and stale-if-error. In my deployments, stale-while-revalidate=60 on an API response has cut origin load by 30% during traffic spikes because the edge can serve a slightly stale response instantly while updating in the background.

# Immutable static asset — fingerprinted filename, cached for a year at edge and browser
HTTP/1.1 200 OK
Content-Type: application/javascript
Cache-Control: public, max-age=31536000, immutable
Content-Length: 18432
ETag: "a3f8c1e9b2d4"

# API response — short browser cache, longer edge cache, serve stale while updating
HTTP/1.1 200 OK
Content-Type: application/json
Cache-Control: public, max-age=60, s-maxage=300, stale-while-revalidate=60, stale-if-error=86400
ETag: "33a64df551425fcc55e4d42a148795d9f25f89d4"

# Private, sensitive data — do not store anywhere
HTTP/1.1 200 OK
Content-Type: text/html
Cache-Control: no-store, private
Pragma: no-cache

Notice the immutable directive on the first example. It is a hint that the content at this URL will never change — ideal for assets with content-hashed filenames like app.9c2e4d.js. Browsers that understand it will not even revalidate during page reloads. As defined in the HTTP Working Group Specifications, immutable is an extension, not a replacement for max-age, so always pair it with a long max-age.

Heuristic Caching and Why You Should Explicitly Disable It

When a response has no Cache-Control or Expires header, caches do not necessarily go straight to the origin on every request. Per RFC 9111, they may apply heuristic caching based on other headers like Last-Modified. A common heuristic is to cache for a fraction — typically 10% — of the time since Last-Modified. A file last modified a year ago could be cached for over a month without you ever asking for it.

This is dangerous. I once debugged an origin that served user-specific HTML without any Cache-Control header but with a Last-Modified timestamp from the underlying template file. The CDN applied heuristic caching and served one user's dashboard to another. The fix was simple: send an explicit Cache-Control: no-store or at least Cache-Control: no-cache for any response that should not be cached.

For API responses that change frequently but where you have not yet defined a proper caching policy, be explicit:

# Explicitly uncacheable API response — prevents heuristic caching
HTTP/1.1 200 OK
Content-Type: application/json
Cache-Control: no-store
ETag: "f2a5c9e1d8b3"

If you want revalidation but no heuristic freshness, use Cache-Control: no-cache or max-age=0, must-revalidate. Both force a conditional request. The difference is subtle: no-cache can carry additional field names like no-cache="Set-Cookie" in older specs, but in practice, treat max-age=0, must-revalidate as “always check the origin, but you may store and revalidate.”

ETag and Last-Modified: Validators That Make Revalidation Cheap

When a cached response becomes stale, the cache does not have to download the entire body again. It can ask the origin: “Has this changed?” using validators. Two validators exist: ETag and Last-Modified. They are sent by the origin on the initial response and echoed back by the cache on the next request.

Strong vs. Weak ETags

An ETag is an opaque identifier for a specific version of a resource. A strong validator like ETag: "33a64df551425fcc55e4d42a148795d9f25f89d4" means byte-for-byte identical content. A weak validator prefixed with W/ like ETag: W/"33a64df551425fcc55e4d42a148795d9f25f89d4" means semantically equivalent but not necessarily identical — useful when you generate ETags with minor, insignificant differences like whitespace or gzipped vs. uncompressed variants.

In my experience, strong ETags are almost always what you want for static assets and API payloads. I generate them as a hash of the response body. Some origins use inode-size-mtime tuples, which breaks when you have multiple servers behind a load balancer with slightly different deploy times. If you run multiple origins, ensure ETags are consistent across nodes or switch to a content hash. I have seen otherwise cacheable assets revalidated constantly because each origin node produced a different ETag.

Last-Modified provides a timestamp validator with one-second resolution. It is weaker than ETag because changes within the same second are invisible, and some file systems only update mtime on a delay. If both validators are present, modern caches prefer ETag because it is more precise.

For dynamic content where computing a hash is expensive, you can use a version column or revision ID from your database as the ETag. I often set ETag: "v48291" where the number is the database row version. It is cheap to compute and perfectly precise for revalidation.

Conditional Requests: How If-None-Match and If-Modified-Since Save Bandwidth

Validators become useful through conditional requests. When a cache has a stale entry with an ETag, it sends If-None-Match with that value. For Last-Modified, it sends If-Modified-Since. The origin then either responds with 304 Not Modified and no body if the resource is unchanged, or a full 200 OK if it has changed.

A 304 response is tiny — typically under 200 bytes — compared to potentially megabytes of body data. Over millions of requests, the bandwidth and origin CPU savings are substantial. The cache then refreshes the freshness lifetime of its stored entry and serves it, often updating headers like Cache-Control if the origin sent new ones in the 304.

What Happens on the Wire

# Initial request — cache miss, full response from origin
GET /api/user/1234/profile HTTP/1.1
Host: api.example.com

HTTP/1.1 200 OK
Cache-Control: public, max-age=60, must-revalidate
ETag: "v48291"
Last-Modified: Wed, 21 Oct 2025 07:28:00 GMT
Content-Type: application/json
Content-Length: 1248
{"id":1234,"name":"Alex", ...}

# 60 seconds later — cache entry is stale, send conditional request
GET /api/user/1234/profile HTTP/1.1
Host: api.example.com
If-None-Match: "v48291"
If-Modified-Since: Wed, 21 Oct 2025 07:28:00 GMT

# Origin: nothing changed, send 304 with updated Cache-Control if needed
HTTP/1.1 304 Not Modified
Cache-Control: public, max-age=60, must-revalidate
ETag: "v48291"
# No body — cache reuses stored body and resets age to 0

There are two important edge cases I have learned to watch for. First, if you send both If-None-Match and If-Modified-Since, most origins evaluate If-None-Match first per the spec. If the ETag matches, they return 304 even if Last-Modified would not match. Second, If-None-Match: * is a special form used for PUT requests to prevent overwriting an existing resource — it is not used in normal GET revalidation.

At the CDN layer, I configure the edge to revalidate asynchronously when possible. With stale-while-revalidate, the first request after expiry triggers a background If-None-Match while the edge immediately serves stale. Users never wait for revalidation latency, and the origin sees far fewer blocking requests. This pattern is covered well in web.dev Articles on caching best practices, which I recommend as a complement to the raw RFCs.

The Vary Header: Correct Content Negotiation Without Cache Poisoning

Vary is the most powerful and most dangerous caching header. It tells caches that the response varies based on the value of one or more request headers, and that the cache key must include those headers. Omit it when needed, and you get cache poisoning. Overuse it, and you fragment your cache hit rate to near zero.

The classic example is compression. If your origin sends Content-Encoding: gzip or br depending on Accept-Encoding, you must send Vary: Accept-Encoding. Without it, a cache might store the gzipped version and serve it to a client that only understands identity, resulting in a garbled page. This is one of the first things I audit on new properties. The interaction between compression formats and caching is subtle enough that it deserves its own analysis — I have covered the trade-offs between different encodings in HTTP Compression: Brotli, Zstandard and Gzip Performance Compared.

Content negotiation for language, format, or device type uses the same mechanism. If you serve different JSON vs. HTML based on Accept, or different languages based on Accept-Language, you need Vary: Accept, Accept-Language. But every value of that header creates a separate cache entry.

The Vary Trap and How to Avoid It

I have seen configurations that send Vary: User-Agent to serve slightly different HTML to mobile clients. With thousands of distinct User-Agent strings, this effectively disables caching — the CDN ends up with a cache hit rate below 10% because almost every request has a unique UA. The better approach is to normalize at the edge: map User-Agent to a small set of buckets like mobile and desktop, or better yet, use Client Hints and Vary: Sec-CH-UA-Mobile which has only two values. If you do need full content negotiation, read Content Negotiation at the CDN: Accept, Vary and Client Hints before you add another value to Vary.

Two rules I apply in production:

1. Only list headers that actually change the response body or critical response headers. If you vary on Accept-Language but always return English, remove it.
2. Prefer normalizing or collapsing header values at the edge. For example, normalize Accept-Encoding to just gzip or br before it enters the cache key, rather than caching every permutation of gzip, deflate, br ordering.

And never send Vary: * unless you truly mean “every request is unique and this must never be served from cache.” It is equivalent to no-store for most caches and is rarely intentional.

Stitching It Together: Caching Strategies for HTML, APIs, and Static Assets

Knowing the headers is not enough. You need a coherent policy per content type. Here is the pattern I deploy on most high-traffic sites:

Fingerprinted Static Assets: Cache Forever, Revalidate Never

Assets with a hash or version in the URL — /static/app.9c2e4d1a.js, /images/logo-a1b2c3.webp — can be cached aggressively because the URL itself changes when the content does. I send Cache-Control: public, max-age=31536000, immutable and an ETag for correctness. The CDN caches for a year, the browser caches for a year, and no revalidation is needed. Purge is handled by deploying a new URL, not by invalidating the old one.

HTML Documents: Short Lifetime, Always Revalidate

HTML that references those fingerprinted assets should have a very short cache lifetime. I typically use Cache-Control: public, max-age=0, must-revalidate, s-maxage=60, stale-while-revalidate=300 for public pages. Browsers revalidate every time, but the edge can serve from cache for 60 seconds and serve stale for another five minutes while revalidating. For authenticated HTML, use Cache-Control: private, no-cache or no-store depending on sensitivity. In my experience, adding ETag to HTML is worthwhile even with max-age=0 — a 304 is still far cheaper than retransmitting a 80KB document.

APIs and JSON: Validate, Don't Expire

For APIs, I favor validation over expiration. Send a short max-age — 30 to 60 seconds for frequently updated data — plus a strong ETag derived from the resource version. Include Vary: Accept, Accept-Encoding only if you actually negotiate those headers at the API level. Many APIs always return JSON regardless of Accept; in that case, do not add Vary: Accept — it just fragments the cache for no reason.

One nuance: if your API uses Authorization headers, most CDNs will not cache the response by default, as authenticated requests are considered private per RFC 9111. You must explicitly send Cache-Control: public or s-maxage to override that, and you should be absolutely certain the response does not contain user-specific data before doing so. I have seen a Cache-Control: public accidentally added to an endpoint that returned the current user's email — the edge happily served it to the next visitor.

Finally, remember that modern extensions like Structured Headers in HTTP: The New RFC 9651 Format Explained are changing how headers like Cache-Control and its successors are parsed. While today you write Cache-Control: max-age=60 as a plain string, the underlying parsing model is moving toward structured fields, which will affect how CDNs and browsers handle caching directives in the future. Keeping your header generation free of quoting errors and whitespace quirks will make that transition smoother.

When I audit a site, I start with a simple waterfall: look at response headers for Cache-Control, ETag, Vary, and Age, then watch the second request for If-None-Match and 304. If any of those are missing where they should be present, you have found your optimization opportunity. The specifications at the IETF Datatracker define the full state machine, but in practice, getting these four headers right covers 95% of the performance gain.

Frequently Asked Questions