Structured Headers in HTTP: The New RFC 9651 Format Explained

For years, parsing HTTP headers meant writing ad-hoc string splitting and hoping every origin, browser, and intermediate proxy agreed on where commas, quotes, and semicolons belonged. If you've ever debugged a Vary or Cache-Control mismatch that only reproduced behind a specific CDN PoP, you know the pain. Structured Headers, initially defined in RFC 8941 and now refined and obsoleted by RFC 9651, finally gives us a shared type system for http headers. In my experience running edge logic that inspects and mutates headers at scale, this is the most important improvement to header handling since the introduction of the Host header. RFC 9651 doesn't add new headers itself — it defines a precise data model and serialization so any new header can be parsed deterministically without regex gymnastics. Understanding it is now essential if you build or operate anything that touches headers at the edge.

Why RFC 9651 Replaces RFC 8941: Corrections That Matter on the Wire

RFC 8941 (often called Structured Fields) was a solid start, but deployment exposed ambiguities. RFC 9651, published in September 2024, is formally titled "Structured Fields" and obsoletes 8941. The changes are not cosmetic; they fix interop failures we saw in production parsers. The core data model is unchanged — three top-level structures (Lists, Dictionaries, and Items) built from typed bare items — but the parsing and serialization algorithms are significantly tightened.

The most impactful correction is around byte sequences and display strings. RFC 8941 defined byte sequences as :base64: with strict base64 requirements that many implementations silently relaxed. RFC 9651 clarifies handling of non-canonical base64, whitespace, and padding, and it formally adds Display Strings (%"...") for human-readable Unicode, which was previously handled via ad-hoc percent-encoding. If you were forwarding Sec-CH-UA or custom device hints through an edge worker, you probably saw garbled characters when an upstream re-encoded them — that's exactly what this fixes.

Another subtle fix involves inner lists and parameter serialization. Under 8941, (a b);param=1 and (a b) ; param=1 were considered equivalent, but serializers differed on whether they emitted the space before the semicolon. At the edge, that one-byte difference broke signature validation for headers like Signature-Input and Accept-Signature where the serialized bytes are signed. I've chased 400s from origin signature verification that traced back to a CDN normalizing that whitespace. RFC 9651 locks down the canonical serialization: no optional whitespace before parameters, parameters sorted where required, and deterministic decimal serialization.

Errata That Broke Real Parsers Before RFC 9651

Three errata in particular caused me operational issues. First, integer range handling: 8941 limited integers to 15 digits, but didn't clearly state whether parsers should reject or clamp larger values. Some Rust and Go parsers rejected, while Fastly's VCL-adjacent parsing coerced. Second, token character validation — * and / were allowed in tokens, which collided with Accept and Content-Type style values and led teams to incorrectly model legacy headers as structured. RFC 9651 explicitly calls out that not every header should become structured, and tightens token definitions to avoid this misuse. Third, decimal precision: 8941 required at most 3 digits after the decimal for items but was vague for dictionary values. RFC 9651 mandates that decimals serialize with no more than 3 fractional digits and that parsers reject more, preventing floating-point drift when a weight like q=0.8001 is round-tripped.

For the full normative text, the canonical source is now IETF Datatracker - RFC 9651, and the working drafts and test suite are maintained at HTTP Working Group Specifications.

The Three Structured Types: Lists, Dictionaries, and Items in RFC 9651

Every structured headers field is defined as one of three containers. This constraint is the whole point — instead of free-form text, you declare upfront whether your header is a List, Dictionary, or Item. The parser then knows exactly what to expect and can fail fast.

A List is an ordered sequence of Items or Inner Lists. An Item is a Bare Item plus optional Parameters. A Dictionary is an ordered map of string keys to Items or Inner Lists, where each member can also carry Parameters. Ordering is preserved, which matters for headers like Priority where the order of u and i is not semantically significant but re-serialization must be stable for signing.

# Structured Dictionary example (e.g., CDN-Cache-Control extensions)
# Wire format:
CDN-Cache-Control: max-age=60, stale-while-revalidate=30, no-store

# Structured List example (e.g., Accept-CH style list of tokens)
Accept-CH: Sec-CH-UA, Sec-CH-UA-Mobile, Sec-CH-UA-Platform

# Structured Item example (e.g., Priority header - single dictionary-like item)
Priority: u=3, i=?0

Items as the Atomic Unit

An Item is always BareItem;param1=value1;param2. The Bare Item is one of: Integer, Decimal, String, Token, Byte Sequence, Boolean, Date, or Display String. Parameters are always a string key with an optional Bare Item value; a parameter without =value implies boolean true (?1). This is powerful at the edge because you can attach metadata without a second header. For example, I've used Cache-Status: "souin"; fwd=miss; fwd-status=200 internally to propagate detailed cache state without inventing comma-separated micro-syntax.

When to Use Dictionaries vs Lists at Design Time

If your header carries named directives where order doesn't matter but extensibility does, use a Dictionary. Cache-Control is the classic example that should have been a Dictionary from the start — max-age, stale-while-revalidate are dictionary keys. If your header carries an ordered set of values where duplicates might be meaningful, use a List. Accept-CH or custom X-Features lists are good candidates. Don't use an Item unless the header is truly a single value, like Content-Length would be if redefined today. In my experience, teams overuse Dictionaries for everything; if the client never looks up by key, a List is simpler to parse and cheaper to serialize in WASM edge workers.

Parsing Structured Fields at the Edge: Bare Items, Parameters, and Canonical Serialization

The performance win from rfc 9651 comes from eliminating backtracking. A compliant parser is a single-pass state machine. At the edge, where you might parse thousands of headers per second per core in VCL, JavaScript Workers, or Rust-based isolates, that determinism keeps p99 low. You no longer need to trim, split on commas outside quotes, or handle q-values with separate logic.

Bare Items have strict delimiting. Strings are always double-quoted with escaping for " and \ only — no single quotes, no backticks. Tokens match [a-zA-Z0-9!#$%&'*+\-.^_`|~]+ and are case-sensitive except where the defining header says otherwise. Integers are limited to 15 digits (-999,999,999,999,999 to 999,999,999,999,999). Booleans are ?1 and ?0, not true/false. Dates are @int Unix timestamps. Byte sequences are :base64:. This strictness is intentional; it lets you validate at the edge and reject malformed origins before they poison cache keys.

// Example: Parsing a Structured Dictionary in an Edge Worker (JavaScript)
// Header: RateLimit: max=100; window=60, remaining=42, reset=@1715600000
import { parseDictionary } from 'structured-field-values';

const headerValue = request.headers.get('RateLimit');
const dict = parseDictionary(headerValue);
// dict.get('max') => { value: 100, params: [['window', 60]] }
// dict.get('remaining') => { value: 42, params: [] }
// dict.get('reset') => { value: Date(1715600000), params: [] }

if (dict.get('remaining').value === 0) {
  // Apply edge throttling without parsing integers manually
  return new Response('Too Many Requests', { status: 429 });
}

Parameter Encoding and the Semicolon Trap

Parameters are where I see the most implementation bugs. The syntax is ;key=value or ;key for boolean true. Keys are always tokens, lowercased on serialization. Values must be Bare Items. There is no quoting of parameter keys, and parameter values that are strings must be quoted. A common mistake is to emit ; filename=foo.html instead of ; filename="foo.html" — the former is a token foo.html, which is invalid because . is allowed but the parser may interpret it differently than a string. Another edge case: duplicate parameter keys must be rejected, not last-wins. If you aggregate headers from multiple origins, deduplicate before merging.

Serialization That Survives CDN Re-serialization

If you sign or hash header values, you must use the canonical serialization algorithm from RFC 9651 Section 4. It defines exactly one valid byte sequence for any structured value. In practice, this means: Strings escape only \ and ", Decimals always include a decimal point with 1-3 fractional digits, Integers never have leading zeros, and Lists join with , (comma + space) while parameters join with ; (no space). I've found that testing your edge serializer against the official test suite (structured-field-tests on GitHub, sourced from the httpwg) catches 90% of interop issues before they reach production. Never hand-roll serialization with template literals — use a compliant library for both parsing and serialization, even in VCL via inline Rust or WASM.

# Canonical serialization matters for signatures
# Origin emits (valid but non-canonical):
Signature-Input: sig1=("date" "content-type");created=1715600000;keyid="my-key"

# RFC 9651 canonical form (note no spaces before ; and sorted params if defined):
Signature-Input: sig1=("date" "content-type");created=1715600000;keyid="my-key"

# If your CDN normalizes to this, the signature was computed over different bytes.
# Always normalize BEFORE signing, and verify the normalized form.

How CDNs and Browsers Actually Emit Structured Headers Today

Structured Headers are not theoretical. Chrome, Firefox, and major CDNs already emit them, and if you ignore them, you misinterpret priority and caching signals.

The most deployed example is Priority (RFC 9218, Extensible Prioritization Scheme). It replaces the complex, connection-level prioritization with a simple structured dictionary: Priority: u=3, i=?0 where u is urgency (0-7, lower is higher priority) and i is incremental (?1 or ?0). Browsers now send this on every request for images, scripts, and fetches. At the edge, I use it to drive origin shielding and tiered cache fill: u=0 fetches get routed to a low-latency origin tier, while u=7 prefetch can be served stale. Without understanding structured dictionaries, you'll treat Priority as an opaque string and miss the signal.

Client Hints are the second major user. Sec-CH-UA, Sec-CH-UA-Mobile, and Sec-CH-UA-Platform are defined as Structured Fields. The related Content Negotiation at the CDN: Accept, Vary and Client Hints explains how Vary interacts with these, but the wire format itself is structured: lists of strings and tokens. The same goes for Accept-CH — it's a List of tokens, not a comma-separated free-for-all. If your edge varies cache on Client Hints, parse them as structured lists or you'll create duplicate cache entries for Sec-CH-UA="Chromium";v="124" vs Sec-CH-UA="Chromium"; v="124" (note the space).

CDN-specific headers are adopting the format quickly. Cache-Status (RFC 9211) is a structured list, CDN-Cache-Control is moving toward structured dictionary semantics, and the RateLimit family (RateLimit-Limit, RateLimit-Remaining, RateLimit-Reset) is defined as structured fields. The IETF also defines Retry-After as a structured item that can be either a date or integer. For caching, this ties directly into how we handle staleness and revalidation, which is covered in depth in HTTP Caching Headers: Cache-Control, ETag, Vary and Conditional Requests — structured parsing lets you extend Cache-Control without breaking legacy intermediaries that skip unknown directives they can't parse as structured parameters.

Transport doesn't change the data model, but it's worth noting that when you move to HTTP/3 and QUIC: Why UDP-Based Transport Changes Everything, header compression switches from HPACK to QPACK. Structured fields compress better because their limited character set and predictable structure produce smaller Huffman encodings. I've measured 8-12% reduction in header bytes on image-heavy pages after moving priority signals to structured form under HTTP/3.

Interoperability Traps When Proxies Strip or Re-Serialize Structured Headers

The biggest operational risk is not parsing — it's re-serialization by intermediaries that claim to support structured headers but don't. A legacy proxy that treats all headers as strings will happily do headers.set('Priority', headers.get('Priority').toLowerCase()) or strip parameters it doesn't recognize. That violates RFC 9651 and breaks downstream consumers.

I hit this with a popular origin shield that stripped unknown parameters from Cache-Status to "sanitize" headers. The edge was emitting Cache-Status: "cloudflare"; hit, "souin"; fwd=miss as a structured list with parameters, but the shield collapsed it to Cache-Status: cloudflare; hit, souin; fwd=miss — removing the quotes and turning strings into tokens. The next hop's parser then rejected it because hit as a token with no value in that position is invalid. Lesson: if you don't understand a structured header, forward it verbatim. Never normalize or case-fold the value.

Another trap is combining headers. HTTP allows multiple header field lines with the same name to be combined with , . For structured fields, that combination is only valid if the structured type is List. Combining two Dictionary headers with a comma produces an invalid dictionary. Some edge runtimes do this automatically when you call headers.append() twice. Check your runtime's behavior — Cloudflare Workers and Fastly Compute preserve separate lines, but Node's http module concatenates.

Use this reference when deciding how to handle a header at the edge:

Finally, watch for intermediaries that add whitespace for "readability." Adding a space after ; in parameters or before , in lists produces non-canonical output. It will still parse, but if the header is covered by Signature-Input, verification fails. In my edge pipelines, I run a validation worker that parses and re-serializes all outgoing structured headers canonically before signing, so any upstream pretty-printing is undone.

Frequently Asked Questions