HTTP/2 Flow Control Tuning for CDN: Window Sizes and Stream Management

Last summer we pushed a new VOD origin through our Amsterdam edge and watched throughput collapse to 40% of what our HTTP/1.1 fleet was delivering on the same links. No packet loss, no CPU spike, just slow. It turned out we'd left HTTP/2 flow control at its defaults while serving 2-4 MB video chunks over a 60ms transatlantic path. That default 64KB window couldn't keep the pipe full, and it took us two days of tcpdump and nghttp2 tracing to prove it. If you're running a CDN, flow control tuning isn't a nice-to-have — it's what decides whether your edge actually uses the bandwidth you're paying for.

The 64KB Trap: Why Default Windows Starve High-BDP CDN Paths

HTTP/2 flow control is hop-by-hop and window-based, and every implementation starts conservative. Per RFC 9113, both the connection window and each stream window begin at 65,535 bytes unless you advertise something larger in the SETTINGS frame. That's fine for a low-latency browser-to-edge connection, but it's brutal for edge-to-origin or edge-to-edge links where the bandwidth-delay product (BDP) is high. I've seen this exact issue bite teams who assume TCP auto-tuning will somehow fix HTTP/2 stalls. It won't. TCP can have megabytes in flight while your HTTP/2 layer is artificially stalled waiting for a WINDOW_UPDATE.

Here's the math we should have done before that Amsterdam launch. On a 1 Gbps link with 60ms RTT, BDP is roughly 7.5 MB. With a 64KB window size, the sender can only have 64KB unacknowledged at the HTTP/2 layer before it must stop, regardless of what TCP allows. That caps your theoretical throughput at window / RTT = 64KB / 0.06s = ~8.7 Mbps per stream. Even with multiplexing, you're sharing the same connection window, so you don't linearly scale your way out. Flow control tuning means sizing those windows to cover BDP, plus headroom for jitter. We now calculate target window as BDP * 1.5 and never deploy an edge with less than 256KB for origin-facing connections.

You set this in two places: the initial window size you advertise to peers via SETTINGS_INITIAL_WINDOW_SIZE, and the dynamic WINDOW_UPDATE frames you send as you consume data. Most proxies like nginx, Envoy, and Apache Traffic Server let you configure the initial value, but you also need to ensure your application consumes buffers quickly enough to send WINDOW_UPDATEs without delay. A slow reader that holds onto 128KB of data will stall the sender just as surely as a small window.

# nginx (edge) - raising initial window for origin fetches
# Requires nginx 1.19.7+ with http_v2_module
http {
    # Connection window - applies to the entire HTTP/2 session
    http2_chunk_size 256k;
    
    # Initial stream window advertised in SETTINGS
    # Tune based on BDP: we use 512KB for transatlantic origins
    http2_initial_window_size 512k;
}

# Envoy example - per RFC 9113 SETTINGS frame
# In envoy.yaml for cluster upstream_http_protocol_options
upstream_http_protocol_options:
  auto_config:
    http2_protocol_options:
      initial_stream_window_size: 524288  # 512KB
      initial_connection_window_size: 1048576 # 1MB - must be >= stream window

Decoupling Connection and Stream Windows for Real CDN Workloads

A mistake I still see in configs is treating connection and stream windows as the same knob. They're not, and tuning them together for all traffic types will hurt you. The connection window throttles every stream multiplexed on that session, while each stream window throttles a single response. For our image-heavy e-commerce customers, we multiplex 80-120 small assets (15-50KB) per page over one connection. For our video customers, we push 2-6 large segments sequentially. Those two patterns need opposite flow control tuning.

For many small streams, you want a large connection window (1-2MB) but you can keep stream windows modest (128-256KB). The large connection window prevents head-of-line blocking at the session level where 50 streams all try to make progress, while small stream windows limit the buffer bloat if a client is slow to consume one response. I've had origins OOM because we set every stream to 1MB and a burst of 100 concurrent slow clients pinned 100MB per connection in proxy buffers. Conversely, for large object delivery, you want both windows large. If your stream window is 1MB but your connection window is 256KB, that single large stream still stalls at 256KB. The HTTP Working Group Specifications are explicit that a sender must respect the minimum of the two windows, but many engineers miss that implication when they only raise one.

This is where HTTP/2 Stream Prioritization at CDN Edges: Delivering Critical Resources First interacts directly with flow control. If you give every stream an equal 512KB window, a low-priority background image can consume connection window budget that should have gone to your critical CSS or hero image. We now tie our stream management to priority signals: high-priority streams get an immediate WINDOW_UPDATE to refill their window up to 512KB, while low-priority streams are artificially held at 64KB until critical streams complete. You can also look ahead to how HTTP/3 Connection Migration: Seamless Handoffs Across Network Changes changes this model, since QUIC's flow control is similar but per-connection migration makes stale window calculations even riskier.

// nghttp2 / Node.js h2 server - dynamic flow control tuning
// We adjust windows based on content type detected at origin

const http2 = require('node:http2');

const server = http2.createSecureServer({
  settings: {
    initialWindowSize: 256 * 1024, // Default for most streams
    maxConcurrentStreams: 100
  }
});

server.on('stream', (stream, headers) => {
  const isLargeAsset = headers[':path'].match(/\.(mp4|ts|zip)$/);
  
  if (isLargeAsset) {
    // Bump this specific stream's window to 1MB without affecting others
    // This sends a WINDOW_UPDATE with increment = 1MB - 64KB
    stream.setWindowSize(1024 * 1024);
    
    // Also ensure connection window is large enough
    // Note: in nghttp2, you must track connection window separately
    const session = stream.session;
    if (session.getWindowSize() < 1024 * 1024) {
      session.setWindowSize(2 * 1024 * 1024);
    }
  }
  // For small assets, we keep the default 256KB to bound memory
  // and rely on fast WINDOW_UPDATEs as data is flushed to client
  
  stream.respond({ ':status': 200 });
  stream.end(data);
});

Stream Management Under Backpressure: Stall, Reset, or Queue?

Even perfectly sized windows will hit backpressure when an edge is overloaded or a client is on a 3G link. How you manage streams at that point defines your tail latency. The naive approach is to let the HTTP/2 flow control stall the sender and hold data in proxy memory until the client sends WINDOW_UPDATE. That's what we did in 2019 and it caused memory pressure that cascaded into 502s across an entire PoP. We had streams stalled for 8-12 seconds holding 512KB each, and with 10k concurrent connections we were burning 5GB of buffer just waiting.

Our current strategy is more opinionated. First, we enforce strict stream concurrency limits per connection and per worker — typically 100 streams max for browsers, 20 for origin fetches. Second, we don't let stalled streams sit forever. If a stream's window stays at zero for more than 2 seconds and its priority is low (images, analytics beacons), we send RST_STREAM with CANCEL and let the client retry on a fresh stream where WebTransport Through CDN Proxies: Bidirectional Streams Over HTTP/3 or a new HTTP/2 connection might have better windows. For high-priority streams, we instead apply application-level backpressure: we stop reading from origin rather than buffering, which propagates the stall backward via TCP flow control instead of exploding our edge memory. The web.dev Articles on optimizing HTTP/2 delivery show similar patterns, but they rarely discuss the memory vs. latency tradeoff at CDN scale.

You'll need to measure this continuously. We export three metrics per PoP: average WINDOW_UPDATE latency (time between data consumption and update sent), stream stall time (time window == 0), and flow-control-limited throughput (% of time sender was window-blocked vs. TCP-blocked). If stall time climbs while TCP utilization is low, your windows are too small. If buffer memory climbs while stall time is low, your windows are too large for your concurrency. There's no single correct size — we retune weekly based on actual BDP distributions.

Putting It Into Practice: Tuning WINDOW_UPDATE at the Edge

You've got the theory — now you've got to ship it without blowing up memory on 50k concurrent connections. The mistake we see teams make is setting both windows to some huge static value like 16MB and calling it done. Don't do that. On a CDN edge with 40k HTTP/2 connections per node, a 16MB connection window is 640GB of potential buffer pressure if your autotuning is naive. You want ceilings that are high enough for your worst BDP path, but with dynamic replenishment that doesn't spam WINDOW_UPDATE every 16KB.

What works in production is decoupled, autotuned windows. Start with a conservative stream window of 1MB (1048576 bytes) and a connection window of 2-4MB, then grow them based on measured RTT and throughput. For a typical US-EU path at 85ms RTT and 1 Gbps, your BDP is roughly 10-11MB — you'll never fill that pipe with a 64KB window, you'll top out around 6 Mbps. We autotune up to 2MB per stream for video and large object delivery, but we keep API JSON streams at 256-512KB. That's also why you should decouple your HTTP Content Coding Negotiation: Accept-Encoding Beyond Gzip and Brotli strategy from your window sizing — a Brotli-compressed 2MB asset might only be 400KB on the wire, so your window accounting should track compressed bytes, not origin bytes.

# Envoy - per-listener HTTP/2 tuning for a CDN edge
http2_protocol_options:
  initial_stream_window_size: 1048576  # 1MB vs default 65535
  initial_connection_window_size: 2097152 # 2MB
  # Envoy autotunes beyond this, but sets the floor

# Nginx (1.25.1+) - global http block
http {
  http2_chunk_size 128k;
  http2_recv_window 1024k;      # stream window
  http2_recv_buffer_size 256k;
}

# Custom Go origin - don't starve the client
import "golang.org/x/net/http2"
tr := &http2.Transport{
  // Allow server to advertise larger window
}
# And on the server side, ensure you aggressively send WINDOW_UPDATE
# once 25-50% of window is consumed, not at 0. That's 2-4 updates
# per MB instead of 16.

Advanced Patterns and When Flow Control Quietly Breaks

Once you've raised the windows, the next failure mode isn't throughput — it's stalls you can't see in your access logs. The classic CDN ghost is a connection window that flatlines while stream windows look healthy. One slow client downloading a 4K segment holds back WINDOW_UPDATE for the connection, and suddenly 199 other multiplexed streams stall even though their own windows are open. If you're multiplexing 100+ streams per connection to the origin, set your connection window to at least 4-8x your stream window and always prioritize WINDOW_UPDATE for the connection over streams. Better yet, cap streams per connection to 50-100 for high-BDP origins. We've dropped p99 TTFB by 30% just by lowering max_concurrent_streams from 250 to 100 on congested PoPs.

Diagnosing this is miserable without the right telemetry. Don't rely on throughput counters alone. You need to log flow control block time — how long a stream was ready to send but blocked on window exhaustion. Envoy's http2.tx_resets and http2.flow_control_paused_frames are gold, and nghttp2's verbose logs will show you exactly when WINDOW_UPDATE is delayed. If you can, capture qlog or Wireshark and filter for http2.window_update. When you see a sawtooth pattern where the window hits zero every 64KB, your tuning didn't apply. When you see a single stream consuming 90% of connection window increments, you've got a noisy-neighbor stream starving the rest. That pairs directly with issues we cover in HTTP Trailers for CDN: Post-Body Checksums, Timing, and Logging — if you're buffering the whole body to calculate a trailer checksum, you're artificially holding flow control windows hostage until the buffer drains.

And don't confuse flow control with congestion control or application backpressure. If your origin app is slow to produce bytes, the TCP congestion window and HTTP/2 flow control window will both look open, but your stream is still stalled. We've seen teams chase window sizes for weeks when the real culprit was a blocking origin read while handling Advanced HTTP Conditional Requests: If-Match, If-Range, and Partial Updates. For range requests especially, if you're doing a 1-byte If-Range revalidation that turns into a 200 OK full-asset send, you'll burst past your tuned window instantly. Queue intelligently: prefer RST_STREAM(NO_ERROR) for client-cancelled ranges over holding them queued, and never queue more than 2x your connection window worth of pending DATA frames.

Frequently Asked Questions