I still remember watching a 10 Gbps edge node in Singapore flatline at 410 Mbps during a cache-fill from origin in Frankfurt. No packet loss, no CPU pressure, congestion control behaving perfectly — yet the link just wouldn't push harder. It took us two days to realize we weren't looking at a network problem at all, but a 30-year-old protocol default quietly capping every single TCP session. If you've ever chased phantom throughput issues across a CDN, you've probably hit the same wall without realizing it.
The 64KB Ceiling That Modern CDNs Still Hit
When TCP was defined in RFC 793, a 16-bit Window field seemed generous. 65,535 bytes was massive for links measured in kilobits per second. Fast-forward to today and that limit is absurd. Any path with a Bandwidth-Delay Product and CDN Tuning: Filling the Pipe Without Overwhelming It larger than 64KB will stall, because the sender simply isn't allowed to keep enough unacknowledged data in flight to fill the pipe. The math is brutal and simple: max throughput = window size / RTT.
I've seen this bite teams who assume kernel defaults will save them. Most Linux distributions ship with window scaling enabled now, but the default receive buffer autotuning limits often clamp the effective window far below what a high-BDP path needs. I've audited PoPs where net.ipv4.tcp_rmem was left at 4K 87380 6291456 — that middle value becomes the default, and the max is only 6MB. On a 150ms trans-Pacific link at 10 Gbps, you need over 180MB to fully utilize the path. We weren't even close, and our throughput graphs proved it. The window never grew, so TCP performance stayed artificially constrained regardless of how much bandwidth we bought.
Window scaling from RFC 7323 doesn't increase the header field — it adds a negotiated shift count during the handshake that lets both sides interpret the 16-bit value as a much larger window. A scale factor of 7 means the advertised window is left-shifted by 7 bits, or multiplied by 128. Without that option exchanged in the initial SYN, you're locked to 64KB for the entire connection lifetime. No amount of mid-connection buffer tuning can fix a handshake that didn't negotiate scaling. Check your SYN captures first, always.
# Check if scaling was actually negotiated - not just enabled
sysctl net.ipv4.tcp_window_scaling
# Should be 1, but that only enables the option
# See the *effective* scale and window in real time
ss -i -t '( dport = :443 )' | grep -E "cwnd|rcv_space|rtt"
# Typical handshake failure looks like this in tcpdump:
# 192.0.2.10.54321 > 203.0.113.50.443: Flags [S], win 65535, options [mss 1460,nop,wscale 7]
# 203.0.113.50.443 > 192.0.2.10.54321: Flags [S.], win 65535, options [mss 1460,nop,wscale 0]
# ^ Second peer didn't scale - connection is now capped at 64KB forever
# Inspect autotuning limits that silently cap you
sysctl net.ipv4.tcp_rmem net.ipv4.tcp_wmem net.core.rmem_max net.core.wmem_maxHow Window Scaling Is Negotiated and Why It Fails Silently
Window scaling is deceptively fragile because it's a one-shot negotiation. Both endpoints must send the Window Scale option (Kind 3) in their SYN and SYN-ACK. The value each sends is the number of bits to shift its own receive window, so the two directions can have asymmetric scale factors. If a middlebox strips TCP options — and I've seen carrier-grade firewalls, older load balancers, and even some DDoS scrubbers do exactly that — the endpoints fall back to unscaled windows without any error. The connection succeeds, data flows, but throughput optimization is dead on arrival. You won't see a reset or a log entry; you'll just see subpar throughput that looks like congestion.
That's why RTT Measurement Methodology for CDN: Getting Accurate Round-Trip Times matters so much before you tune anything. Your required window is directly proportional to RTT, and if your RTT measurements are smoothed, averaged, or sampled from the wrong percentile, your window calculations will be wrong. I learned this debugging a South America PoP where our SYN-ACK RTT was 35ms but our p95 data-ACK RTT was 110ms due to bufferbloat on last-mile links. We'd sized windows for 35ms and wondered why rebuffering persisted during peak. Once we measured correctly and sized for the real distribution described in CDN Percentile Latency Analysis: What p50, p95, and p99 Actually Tell You, the required window tripled.
Another trap is assuming window scaling alone fixes things. It just raises the ceiling; autotuning and socket buffers must be allowed to grow into it. On CDN origins and edges handling thousands of concurrent connections, memory pressure makes operators nervous about raising tcp_rmem, but capping it too low defeats scaling entirely. The kernel will advertise a smaller window than the scale factor allows if it doesn't have buffer space to back it. For standards details on how scaling interacts with flow control, the HTTP Working Group Specifications and TCP extensions are essential reading, and web.dev Articles provide solid context on how browser-side socket limits compound the problem.
# Capture the handshake and confirm scale factors in both directions
tcpdump -i eth0 'tcp[tcpflags] & tcp-syn != 0' -vv -n | grep -i wscale
# Example good negotiation (both sides scale):
# localhost.5201 > peer.443: Flags [S], win 64240, options [wscale 7,mss 1460,sackOK]
# peer.443 > localhost.5201: Flags [S.], win 65535, options [wscale 8,mss 1460,sackOK]
# Effective max windows: 64240*128 = ~8 MB client, 65535*256 = ~16 MB server
# Force a throughput test that will expose window limits (not just cwnd)
iperf3 -c 203.0.113.50 -P 1 -w 4M -t 30 -R # -w sets socket buffer, -R reverses direction
# Compare with: iperf3 -c 203.0.113.50 -P 1 -w 64K -t 30 -R
# If the 4M result is 5-10x higher, you were window-limited, not congestion-limited
# Watch actual advertised window in-flight (Linux)
cat /proc/net/tcp | awk '{print $2,$7,$A}' # Or use `ss -e` for window scaling factorWhat We Measured When Defaults Were Left Untouched
Earlier this year we ran a controlled set of cache-fill tests between Ashburn and Singapore to isolate window size from congestion control. Same BBRv2, same 10 Gbps NICs, same loss rate (<0.01%), only the advertised window changed. We used a 100ms emulated RTT with netem to keep conditions consistent and drove single-stream HTTPS fetches for 2GB objects. With scaling disabled — simulating a stripped SYN option — throughput hit a hard wall regardless of how much we tuned congestion windows. Enabling scaling with sane buffer maxima removed that wall entirely.
The throughput numbers weren't subtle. At 100ms, a 64KB window mathematically caps you at 5.24 Mbps (65535 bytes * 8 / 0.1s). Even with window scaling enabled but tcp_rmem capped at 256KB, we couldn't sustain more than ~20 Mbps per stream. That means to fill a 1 Gbps link you'd need 50 parallel connections, which kills TTFB and origin efficiency. When we raised net.core.rmem_max and tcp_rmem to allow a 4MB window, a single connection sustained over 310 Mbps on the same path. For CDN throughput, that single-stream performance is everything — video segment delivery, large object caching, and prefetch all suffer when the window is too small.
Defaults fall short because they were chosen for host memory safety in the 2000s, not for 2025's high-BDP edge. If you're operating PoPs with diverse RTTs, you can't pick one window size. You need autotuning with generous maxima and you must actively monitor the actual advertised window, not just cwnd. We've now added window scaling negotiation success rate to our edge health checks alongside loss and retransmits. When that rate dips below 99.5%, we know a middlebox or bad config is silently throttling us.
| Test Scenario (Ashburn → Singapore, 100ms RTT) | Advertised Window | Theoretical Max Throughput | Measured Single-Stream Throughput | Connections Needed for 1 Gbps |
|---|---|---|---|---|
| Unscaled (wscale 0), 64KB | 64 KB | 5.24 Mbps | 5.1 Mbps | 196 |
| Scaled w/ default buffers, 256KB cap | 256 KB | 20.97 Mbps | 19.8 Mbps | 51 |
| Scaled w/ tuned buffers, 1MB window | 1,024 KB | 83.89 Mbps | 81.4 Mbps | 13 |
| Scaled w/ tuned buffers, 4MB window | 4,096 KB | 335.54 Mbps | 312.7 Mbps | 4 |
| Scaled w/ tuned buffers, 16MB window (BDP for 10Gbps @ 100ms) | 16,384 KB | 1,342.18 Mbps | 1,180.5 Mbps* | 1 |
Tuning Window Scaling for High-Throughput CDN Edges
Leaving window scaling on its defaults is effectively letting the OS guess your workload, and it’ll guess wrong. On most Linux distributions we see in the wild — Ubuntu 22.04, ALMA 9, even tuned Amazon Linux 2023 — `net.ipv4.tcp_window_scaling` is enabled (1), but `net.core.rmem_max` and `net.core.wmem_max` are still capped at 212992 bytes, and `net.ipv4.tcp_rmem` sits at `4096 131072 6291456`. That third value is the theoretical max, but the default advertised window during slow-start rarely scales past 64-128KB unless memory pressure is low and the autotuning algorithm actually ramps. For a PoP serving 4K video segments over a 80ms transatlantic RTT, that caps you at around 8-12 Mbps per flow. You’ll never saturate a 25 Gbps NIC with that.
We’ve standardized on a much more aggressive autotuning profile for all cache and origin-shield tiers. You don’t want to hardcode a massive window — you want to let the kernel scale it dynamically but give it headroom to do so. That means raising the max buffers while keeping the mins sane so you don’t OOM under 100k concurrent connections. It’s a balance between CDN Throughput vs Bandwidth: Understanding the Difference and Why It Matters — bandwidth is what the link offers, throughput is what TCP will actually let you use. On a lossy, high-BDP path, a 2MB window isn’t wasteful, it’s mandatory.
# /etc/sysctl.d/99-cdn-window-scaling.conf
# Enable scaling and give autotuning room to breathe
net.ipv4.tcp_window_scaling = 1
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.rmem_default = 262144
net.core.wmem_default = 262144
net.ipv4.tcp_rmem = 4096 262144 16777216
net.ipv4.tcp_wmem = 4096 262144 16777216
net.ipv4.tcp_moderate_rcvbuf = 1
# Apply and verify negotiated scale factor
sysctl -p /etc/sysctl.d/99-cdn-window-scaling.conf
ss -i -t | grep -i "wscale"
# Expected: wscale:7,7 (shift count 7 = 128x scaling, 64KB -> 8MB)When Window Scaling Silently Breaks: Diagnostics You Can't Skip
The nasty part isn’t that window scaling fails — it’s that it fails silently and looks like congestion. We debugged a case last quarter where a Tier-1 ISP’s CGNAT was stripping the WSOpt (kind=3) from SYN packets. The handshake completed, the connection established, but both sides fell back to a 64KB unscaled window. Throughput for that entire ASN flatlined at ~5.2 Mbps on our 60ms RTT paths, while latency and loss looked pristine. Standard dashboards showed green. It wasn’t until we ran `tcpdump 'tcp[tcpflags] & tcp-syn != 0'` on the edge and saw SYN-ACKs coming back without the wscale option that we knew. If you’re not actively monitoring the negotiated shift count via `ss -i` or eBPF, you’re blind to this.
Another pattern we see constantly interacts with jitter and reordering. A large scaled window is fantastic until you pair it with a shallow buffer on an intermediate router. When you advertise an 8MB window, you’re inviting the sender to have 8MB in flight. If that burst hits a congested hop that can’t handle it, you don’t get graceful throttling — you get burst loss, SACK storms, and retransmissions that crush goodput. This is where CDN Jitter Measurement and Its Impact on User Experience becomes critical, because high variance in RTT makes BBR and CUBIC misestimate the BDP and overshoot the window. We mitigate this by capping the max window per-route class — 4MB for mobile eyeball networks, up to 16MB for shield-to-shield backbone — rather than using one global value.
And don’t ignore the receiver-side interaction with packet reordering. We used to chase what we thought was a window scaling issue, only to find it was actually Packet Reordering Effects on CDN Performance: The Hidden Throughput Killer triggering spurious fast retransmits that collapsed the congestion window, which then prevented the receive window from ever growing. Your `tcpdump` will show duplicate ACKs and shrinking windows, but the root cause isn’t the window at all — it’s reordering causing the sender to think the path is congested. Always correlate `TCPRetransFail`, `TCPDSACKRecv`, and `TCPLoss` counters from `nstat` before you blame the scale factor. A fixed window won’t help if the stack thinks it needs to be conservative.
Frequently Asked Questions
What's the ideal window scaling shift count for modern CDN workloads?
There's no single ideal, but for most CDNs we recommend negotiating a shift count of 7 or 8. A shift of 7 multiplies the 16-bit window by 128, giving you a max window of ~8MB (65535 * 128). That's enough to saturate a 10 Gbps link up to ~6-7ms RTT, or a 1 Gbps link up to ~65ms. A shift of 10+ gives you up to 1GB windows, but you rarely need that and it can mask bufferbloat issues. Check your actual BDP: Required Window = Bandwidth (bytes/s) * RTT (seconds).
Does enabling window scaling increase memory usage on edge servers?
Yes, but not as much as you'd fear if you use autotuning correctly. The window is an advertised limit, not a pre-allocation. Memory is only consumed as data is actually buffered in-flight. With `tcp_moderate_rcvbuf=1`, the kernel grows the buffer from the default (e.g., 256KB) only as the sender utilizes it. Under our production load of ~45k concurrent connections per node, moving from 212KB to 16MB max increased average per-socket memory from ~92KB to ~210KB — an extra ~5.3GB per server, but throughput for long-lived connections improved by 340%.
Can middleboxes or firewalls block window scaling?
Absolutely, and they do it more often than you'd expect. Older firewalls, some AWS NLB configurations, and carrier-grade NATs are notorious for stripping unknown TCP options or rewriting the window without rescaling it correctly. The symptom is a connection that establishes fine but never exceeds ~64KB in flight despite both hosts supporting scaling. Always validate with packet captures on both sides of the suspected middlebox and monitor the `wscale` value reported by `ss -i` in production.
How does window scaling interact with TCP BBR versus CUBIC?
With CUBIC, the receive window is often the hard ceiling — CUBIC will keep probing for bandwidth until it hits loss, so a small window directly caps it. BBR is different; it models the BDP and tries not to fill the window unnecessarily, so it's less likely to be window-limited but more sensitive to jitter in its RTT probe. In practice, we've found BBR benefits even more from correctly tuned scaling because when BBR does decide to send a burst (like during `PROBE_BW`), it needs that large window available immediately, or it'll incorrectly estimate the bottleneck bandwidth as lower than it really is.