Content Negotiation at the CDN: Accept, Vary and Client Hints

Content negotiation is one of those HTTP features that looks simple in the RFC and becomes painfully complex once a CDN sits between the client and origin. On paper, the client sends what it prefers, the server responds with the best match, and caches store variants. In production at the edge, I've seen negotiation silently destroy cache hit ratios, serve Brotli to clients that can't decode it, and leak personalized language variants to the wrong users. Getting it right means understanding not just the Accept-* family of headers, but how the Vary response header, cache keys, and the newer Client Hints mechanism interact when every request is evaluated in under a millisecond at 300+ PoPs.

How Proactive Content Negotiation Actually Works at the Edge

In my experience, most developers think of negotiation as an origin concern. The reality is that any caching layer that doesn't participate correctly will either break negotiation or make it useless. Proactive negotiation — where the server selects a representation based on request headers without a round-trip — is the only form that works efficiently through a CDN. The client sends headers like Accept, Accept-Language, and Accept-Encoding, and the server is expected to choose.

The critical detail is that the CDN must be part of that selection, or at minimum, must cache each variant correctly. Per the HTTP Working Group Specifications (RFC 9110 Section 12), a server that performs proactive negotiation SHOULD generate a Vary header field listing the request headers that influenced the selection. If your origin does this but your CDN ignores it, you have a correctness bug. If your origin doesn't send it, you have a different correctness bug.

Where Selection Should Happen: Origin vs. Edge

There are two viable architectures. In the first, the origin performs selection and the edge simply respects Vary to partition the cache. This is simple and correct, but it punts all the logic upstream and adds latency on cache misses. In the second, the edge performs selection using Edge Workers, VCL, or native image optimization features. I've found that second model essential for formats like Accept: image/avif,image/webp where you want to serve AVIF from cache without ever hitting origin for a negotiation decision.

What does not work is letting the CDN cache a single representation and serve it to everyone regardless of request headers. I've debugged production incidents where an image CDN cached the WebP variant first, omitted Vary: Accept, and then served WebP bytes with Content-Type: image/webp to a legacy client that only advertised image/jpeg. The browser showed a broken image and no amount of purging fixed it until the Vary behavior was corrected.

The Quality Value (q-factor) Trap

Clients can send weighted preferences: Accept-Language: fr-CH, fr;q=0.9, en;q=0.8, de;q=0.7, *;q=0.5. Many origin frameworks parse this correctly. Very few edge implementations do. If you are writing negotiation logic in an Edge Worker, don't implement naive string matching. Parse the q-values, sort by weight, and handle the wildcard explicitly. Otherwise you will serve German to a user who explicitly preferred French-Swiss.

The Vary Header: Why It Controls Your Cache Hit Ratio

Vary is the most misunderstood caching header after Cache-Control. It doesn't control the browser. It tells any shared cache — including your CDN — that the response is only valid for requests that have the same values for the listed request headers. For a deeper look at how it fits with other cache validators, see HTTP Caching Headers: Cache-Control, ETag, Vary and Conditional Requests.

A response of Vary: Accept-Encoding means gzip and Brotli variants are cached as separate objects under the same URL. A response of Vary: Accept-Encoding, Accept-Language means the cache key is effectively URL + Accept-Encoding value + Accept-Language value. Every header you add to Vary multiplies your cache variants.

HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Content-Encoding: br
Content-Language: en
Vary: Accept-Encoding, Accept-Language
Cache-Control: public, max-age=86400

# CDN cache key becomes (conceptually):
# /index.html | br | en  -> cached object 1
# /index.html | gzip | en -> cached object 2
# /index.html | br | fr  -> cached object 3

Vary Star and Why You Should Never Use It

Vary: * tells a cache that every request header could have influenced the response, which per RFC 9111 means the response is essentially uncacheable by shared caches. I have seen this header accidentally emitted by application servers when negotiation logic encounters an unknown header. If you see Vary: * in your edge logs, treat it as a P0. Your hit ratio for that object will drop to zero.

When Vary Lies

A more subtle problem is a lying Vary. The origin sends Vary: Accept-Encoding but also varies on User-Agent implicitly for image format without declaring it. The CDN therefore collapses all User-Agents into one cache entry and serves the wrong format. I've found that auditing origin responses with curl -H "Accept: image/avif" -I versus curl -H "Accept: image/webp" -I and comparing Vary is the fastest way to catch this. If the bytes change but Vary doesn't list the header that caused the change, the cache is incorrect.

Accept, Accept-Language and Accept-Encoding Handling at the Edge

These three headers have very different characteristics at the edge, and treating them the same is a mistake. Accept-Encoding is highly cacheable because it has low cardinality — typically just gzip, br, and occasionally zstd. Accept for images has moderate cardinality but predictable values. Accept-Language has enormous cardinality and is the primary reason for cache fragmentation.

For compression specifically, the interaction between negotiation and content encoding deserves careful attention. Our testing shows Brotli saves 15-20% over gzip for text, but only if the edge doesn't serve a cached gzip object to a Brotli-capable client due to a collapsed cache key. For a detailed comparison of the algorithms themselves, see HTTP Compression: Brotli, Zstandard and Gzip Performance Compared.

Normalizing Accept-Encoding Before It Hits Cache

Clients send messy values: gzip, deflate, br vs br, gzip vs gzip;q=1.0, br;q=0.8. If you use the raw header value as part of the cache key, you create duplicate entries for the same logical encoding. At the edge, normalize first.

// Edge Worker - normalize Accept-Encoding to a canonical value
function normalizeEncoding(header) {
  if (!header) return 'identity';
  const enc = header.toLowerCase();
  // Prioritize Brotli if client supports it and we can serve it
  if (enc.includes('br')) return 'br';
  if (enc.includes('zstd')) return 'zstd';
  if (enc.includes('gzip')) return 'gzip';
  return 'identity';
}

// Then use normalized value for cache key and origin fetch
// Cache key: url + '|' + normalizeEncoding(request.headers.get('Accept-Encoding'))
// Forward to origin: Accept-Encoding: br (or gzip if br not supported)

This approach reduces variants from potentially dozens to three, and it lets you control what you advertise to origin. I've found that collapsing deflate into gzip and stripping q-values for encoding is safe in practice — no modern browser requires deflate alone.

Client Hints as the Successor to Vary-Based Negotiation

Client Hints were designed to solve the exact explosion problem that Accept-Language and User-Agent negotiation created. Instead of the client proactively sending high-entropy preferences on every request, the server delegates which hints it wants via Accept-CH or the newer Permissions-Policy with Critical-CH. The browser then sends only those hints on subsequent requests.

The headers look like this: Sec-CH-DPR: 2, Sec-CH-Viewport-Width: 1280, Sec-CH-UA-Platform: "macOS", ECT: 4g. They are structured headers per RFC 9651, which matters for parsing at the edge — see Structured Headers in HTTP: The New RFC 9651 Format Explained for the exact serialization rules.

Delegating Hints Without Breaking Cache

The CDN workflow for Client Hints has three steps. First, on the initial navigation, the origin or edge sends Accept-CH: Sec-CH-DPR, Sec-CH-Viewport-Width, Sec-CH-UA-Mobile and ideally Critical-CH: Sec-CH-DPR to force a restart if the hint is critical for above-the-fold images. Second, on the next request the browser resends with those headers. Third, the edge selects the right image width/DPR variant and caches it with Vary: Sec-CH-DPR, Sec-CH-Viewport-Width.

HTTP/1.1 200 OK
Accept-CH: Sec-CH-DPR, Sec-CH-Viewport-Width
Critical-CH: Sec-CH-DPR
Vary: Sec-CH-DPR, Sec-CH-Viewport-Width
Content-Type: image/jpeg
Content-DPR: 2

# Subsequent request from browser:
GET /hero.jpg HTTP/1.1
Sec-CH-DPR: 2
Sec-CH-Viewport-Width: 800
Viewport-Width: 800  # deprecated but still seen

# Edge selects 1600w variant: 800 * 2 = 1600 logical pixels

In my experience, the biggest operational mistake with Client Hints is adding every hint to Vary. If you Vary on Sec-CH-UA-Full-Version, you will fragment your cache by browser build number. Only Vary on hints that materially change the bytes. For responsive images, that's typically DPR and Viewport-Width. For capability detection, it might be none — you can use the hint to decide whether to send AVIF, but still Vary: Accept rather than the hint itself if the image bytes are already selected by Accept.

Privacy and the High-Entropy Boundary

High-entropy hints like Sec-CH-UA-Full-Version or Sec-CH-UA-Full-Version-List require a delegation and are gated behind permissions policy. Low-entropy hints like Sec-CH-UA-Mobile and DPR are sent more freely. At the edge, be careful not to log high-entropy hints in plaintext access logs without considering GDPR and data minimization. I've worked with teams that accidentally stored full User-Agent version lists in edge logs that were retained for 90 days, creating an unnecessary privacy liability.

Normalizing Headers and Designing Cache Keys That Don't Explode

The core tension in negotiation at the CDN is correctness versus hit ratio. Correctness demands you Vary on everything that changes the response. Hit ratio demands you Vary on as little as possible. The compromise is normalization: reducing the infinite space of header values to a small, discrete set before they become part of the cache key.

For Accept-Language, normalization means mapping fr-CH, fr;q=0.9, en;q=0.8 to just fr or en based on your supported languages. Do not use the raw header as a cache key. A site that supports three languages should have at most three variants, not thousands. The best way to achieve this is to avoid Vary: Accept-Language entirely and instead use URL-based negotiation — example.com/en/article and example.com/fr/article — with a 302 at the edge for the language root based on a normalized Accept-Language check.

For Accept image negotiation, normalization means collapsing the long list of MIME types into a format decision. If the client sends image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8, the edge logic should reduce that to avif if you have an AVIF variant, otherwise webp, otherwise jpeg. Cache only three objects, not one per unique header string.

VCL Example for Format Selection

At the edge, Varnish/Fastly-style VCL remains the most deterministic place to implement this. The logic runs before the cache lookup, so you can rewrite the request to a canonical cache key.

sub vcl_recv {
    # Normalize image format negotiation
    if (req.http.Accept ~ "image/avif") {
        set req.http.X-Image-Format = "avif";
    } else if (req.http.Accept ~ "image/webp") {
        set req.http.X-Image-Format = "webp";
    } else {
        set req.http.X-Image-Format = "jpeg";
    }
    # Use normalized header for cache key, not raw Accept
    # hash_data(req.http.X-Image-Format) in vcl_hash
}

sub vcl_backend_response {
    # Ensure origin response advertises correct Vary
    # If we did format selection at edge, we must still signal it
    if (beresp.http.Vary !~ "Accept") {
        if (beresp.http.Vary) {
            set beresp.http.Vary = beresp.http.Vary + ", Accept";
        } else {
            set beresp.http.Vary = "Accept";
        }
    }
}

Notice the second block. Even if you perform selection at the edge, you must emit Vary: Accept downstream. Otherwise a downstream cache — like the browser's HTTP cache — won't know to partition correctly, and a user who navigates back may see the wrong format from disk cache.

Debugging Negotiation Failures at the Edge

When negotiation breaks, the symptoms are often intermittent and hard to reproduce because they depend on which PoP the user hits and what was cached there first. I've found a systematic approach saves hours.

First, check what is actually cached. Purge the URL, then fetch it twice with different headers and inspect Age and X-Cache on the second fetch. If the second request with a different Accept returns HIT but wrong Content-Type, your Vary is missing. If it returns MISS when it should be HIT — for example, two clients both supporting br but sending headers in different order — your cache key is not normalized.

Second, validate that compressed and uncompressed cache keys are isolated. A classic bug is serving a Brotli-compressed body with Content-Encoding: br to a client that doesn't support it, because a shared cache didn't respect Vary: Accept-Encoding. Modern CDNs handle this automatically if you enable compression at the edge, but if you do compression at origin and pass through bytes unchanged, you are responsible for ensuring Vary is correct. Tools like curl --compressed -H "Accept-Encoding: gzip" -I https://example.com/app.js versus curl -H "Accept-Encoding: identity" -I https://example.com/app.js will surface mismatches quickly.

Third, inspect 103 Early Hints interactions. If you use Early Hints to preload the negotiated variant — for example, </hero.avif>; rel=preload; as=image — make sure the hint matches the final negotiation result. I once saw a CDN send an Early Hint for the WebP variant while the final response selected AVIF due to a later Accept evaluation, causing the browser to preload an image it never used. Keep negotiation logic consistent between the Early Hints path and the final response path. The web.dev Articles on Client Hints and Early Hints cover this race in detail.

Finally, log the negotiation decision. Add a response header like X-Negotiated-Format: avif or X-Cache-Variant: br;en during debugging. This makes it trivial to correlate edge logs with user reports and to verify that your Vary matches your selection logic. Remove or strip that header at the outer edge before it reaches the browser if you consider it internal.

Frequently Asked Questions