WAN Optimization Techniques for CDN: Accelerating Long-Haul Connections

Back in 2019, we were pushing a live sports feed from our origin in Frankfurt to an edge PoP in Sydney, about 290ms round-trip, and users were complaining about rebuffering even though we had plenty of capacity. The link wasn't congested, the servers weren't hot, but throughput for a single TCP connection wouldn’t crack 25 Mbps on a 10 Gbps pipe. That’s when it really clicked for me: on long-haul CDN paths, distance itself is the bottleneck, and without deliberate WAN optimization, you’re leaving most of your backbone on the table.

Why Standard TCP Collapses Over Long-Haul CDN Links

Most folks assume TCP just works, and on a LAN it mostly does. But bring in 150-300ms of RTT between continents and the math turns ugly. TCP’s throughput is fundamentally capped by the receive window divided by RTT, and even with window scaling, classic loss-based congestion control like CUBIC misinterprets tiny amounts of loss or jitter as congestion. I’ve seen this repeatedly on our transatlantic links — a 0.1% packet loss event that’d be invisible in a data center causes CUBIC to halve its congestion window, and then it takes dozens of RTTs to ramp back up. On a 200ms path, that recovery is painfully slow, and your CDN WAN link sits underutilized.

We ran into this hard when we expanded to South America from our US-East core. Our initial design relied on default Linux TCP settings and ECMP Routing in CDN Networks: Equal-Cost Multi-Path Traffic Distribution for load sharing. Throughput tests looked great in the lab at 20ms, but in production between Miami and São Paulo (around 110ms), single-stream transfers averaged 40% of line rate. The problem wasn’t routing. It was TCP’s inability to keep the pipe full. We had to stop thinking about bandwidth and start thinking about the bandwidth-delay product (BDP). For a 10 Gbps link at 200ms RTT, the BDP is about 250 MB — that’s how much data must be in flight to fully utilize the link. If your windows, buffers, and congestion control aren’t tuned for that, you’ll never fill it.

That’s where long-haul acceleration starts. The fixes aren’t magic — they’re deliberate tuning of buffers, congestion algorithms, and connection handling. For anyone running a CDN WAN, the first step is always to baseline your BDP and tune host stacks accordingly. Modern stacks give you far better tools than we had a decade ago, but you’ve got to apply them. The HTTP Working Group Specifications have tightened up how large windows and new congestion signals should be handled, and it’s worth reading them alongside the stack docs. Here’s the baseline sysctl profile we now deploy on every backbone cache that handles inter-PoP replication:

# /etc/sysctl.d/99-cdn-wan.conf - tuned for 10Gbps / 100-300ms BDP
net.core.rmem_max = 67108864
net.core.wmem_max = 67108864
net.ipv4.tcp_rmem = 4096 87380 67108864
net.ipv4.tcp_wmem = 4096 65536 67108864
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_sack = 1
net.core.netdev_max_backlog = 250000
net.ipv4.tcp_congestion_control = bbr
net.ipv4.tcp_slow_start_after_idle = 0

# Verify: sysctl -p && ss -i | grep bbr
# Check retrans and cwnd: cat /proc/net/netstat | grep TcpExt

TCP Optimization Techniques That Actually Scale Across Continents

Switching from CUBIC to BBR was the single biggest win we’ve had for CDN WAN performance. CUBIC is reactive — it needs loss to find the ceiling. BBR is model-based; it estimates bottleneck bandwidth and RTT and paces packets to match, so it doesn’t collapse on random loss and it keeps the queue shallow. When we rolled BBRv2 to our backbone nodes handling US to APAC replication, our average single-flow throughput jumped from ~1.2 Gbps to ~4.5 Gbps on the same 180ms path, with 60% lower retransmits. It wasn’t a config change that required new hardware — just a kernel upgrade and tuning.

Beyond the congestion algorithm, you’ve got to address connection setup and slow start. On long-haul paths, the three-way handshake plus TLS negotiation can add 600ms before the first byte of content moves. For CDN origin-to-edge fills, we use persistent, warmed connections with TCP Fast Open (TFO) where we can, and we’ve aggressively increased initial congestion window (initcwnd) from the default 10 to 40-50. I was hesitant to crank initcwnd at first — I worried we’d burst and cause loss — but with BBR pacing the burst, it’s been stable and it shaves a full RTT off small object fetches. We also use connection pooling between PoPs so we’re not constantly paying that handshake tax. If you’re using ATS or Nginx as your mid-tier, pooling is straightforward, and the web.dev Articles on connection coalescing and early hints explain why reusing connections matters even more on high-RTT paths.

The other piece is parallelization. No matter how well you tune a single TCP flow, many WAN optimizers will stripe a large transfer across multiple sockets or use application-level chunking. We do this for cache fills over 10MB. Instead of one 250MB-window flow, we’ll fetch the object in 4x 8MB ranged chunks over four connections. That approach sidesteps per-flow window limits and lets Segment Routing for CDN Traffic Engineering: Source-Routed Path Selection do its job more effectively, since each flow can be load-balanced more granularly. Here’s a simplified example of how we configure ATS to pool and parallelize mid-tier fetches:

# records.config - Apache Traffic Server mid-tier WAN tuning
CONFIG proxy.config.http.server_ports STRING 8080
CONFIG proxy.config.http.keep_alive_enabled_in INT 1
CONFIG proxy.config.http.keep_alive_post_outbound_enabled INT 1
CONFIG proxy.config.net.sock_option_flag_out STRING fastopen
CONFIG proxy.config.http.origin_max_connections INT 200
CONFIG proxy.config.http.cache.http.keepalive INT 1

# Slice large objects for parallel WAN fetch (requires slice plugin)
# plugin.config:
# slice.so --blockbytes=8388608 --remap-host=origin.example.com
CONFIG proxy.config.cache.slice.enabled INT 1

# Verify pooled connections:
# traffic_ctl metric get proxy.process.http.origin_connections_current
# traffic_ctl metric get proxy.process.cache.slice.blocks_fetched

Choosing the Right WAN Optimization Stack for Your CDN Backbone

Not every technique fits every path, and I’ve learned the hard way that throwing every optimizer at a link can make things worse. Deduplication and compression, for instance, look great on paper for enterprise WANs, but on a CDN WAN where most payloads are already compressed images, video segments, or encrypted TLS traffic, the CPU cost outweighs the gains. We tested a commercial deduplication appliance on our EU backbone in 2021 and saw less than 3% reduction on video traffic but a 15% increase in latency from the extra processing. We pulled it after two weeks. For CDN WANs, TCP optimization, intelligent pacing, and smart routing deliver far more than payload compression.

What does work consistently is a combination of BBR-based TCP tuning, forward error correction (FEC) or selective retransmission for lossy long-haul segments, and application-aware routing. On our MPLS-backed backbone, we now pair BBR with per-PoP buffer sizing and we use segment routing to steer replication traffic away from high-loss submarine cable segments when loss exceeds thresholds. For edge-to-origin paths that cross the public internet rather than private backbone, we add a lightweight FEC layer — essentially sending parity packets to recover from 1-2% loss without a retransmission RTT. It adds about 5% overhead, but when you’re saving a 200ms retransmission on a live segment, that’s a worthwhile trade.

Putting WAN Optimization Into Production: A Deployment Blueprint That Holds Up

We’ve seen too many teams lab-test BBR or window scaling and then push it straight to 200 PoPs. Don’t. Long-haul tuning only works when it’s tied to your actual path characteristics — and those characteristics aren’t uniform. A Singapore-to-Frankfurt backbone at 165ms RTT with 0.1% loss behaves nothing like an Ashburn-to-São Paulo link at 135ms but with 2-3% peak loss during congestion windows. Start by segmenting your backbone into RTT/loss buckets and applying profiles, not a global sysctl.

In practice, we’ll deploy a three-tier profile set: Sub-80ms (regional mesh), 80-160ms (intercontinental primary), and 160ms+ (long-haul/satellite backup). For the 80-160ms tier, which carries about 70% of origin-to-edge fill traffic, we’ve consistently seen a 40-55% throughput uplift just by switching from CUBIC to BBR v2, raising initial congestion window to 32-64 MSS, and enabling TCP window scaling with a 1MB+ buffer. Pair that with byte-level caching and LZ4 compression on repetitive manifests and you’ll cut fill time for a 50MB video chunk from 2.1s down to 0.85s on a 150ms path. It’s not magic, it’s just letting TCP actually use the pipe.

The orchestration layer matters as much as the TCP knobs. If you’re still pushing configs via static Ansible playbooks, you’re already behind. This is where Software-Defined Networking for CDN: Programmable Traffic Control pays for itself — we use SDN controllers to push per-path MTU discovery, ECN marking policies, and dynamic FEC ratios in real time. And don’t forget virtualization overhead. When your WAN optimizers run as VMs, you need to account for vSwitch latency and noisy-neighbor contention, which is exactly why we recommend reviewing Network Function Virtualization in CDN Infrastructure: Virtual Appliances at Scale before you virtualize that optimization stack.

# Production-tested sysctl profile for 120-180ms CDN backbone (BBRv2)
# Tested on Ubuntu 22.04 / kernel 5.19+ with 10Gbps backbone NICs

net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_congestion_control = bbr
net.ipv4.tcp_mtu_probing = 1
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_slow_start_after_idle = 0
net.ipv4.tcp_notsent_lowat = 16384

# Verify: ss -tie | grep bbr && nstat | grep TcpExtTCPBBR*

Advanced Patterns and Why Your Long-Haul Link Still Stalls

So you’ve deployed BBR, enabled jumbo frames, and you’re still seeing throughput collapse to 15-20% of link capacity at peak? Nine times out of ten, it’s not TCP — it’s bufferbloat or middlebox interference. We audited a US-EU backbone last quarter where throughput flatlined at 3.2 Gbps on a 10 Gbps circuit. BBR was working correctly, but a legacy firewall in the path was re-segmenting 9000-byte jumbos back to 1500-byte MTUs and holding 800ms of buffer. The fix wasn’t more tuning; it was removing that device from the fast path and letting the SDN fabric handle stateful filtering instead. Always validate end-to-end with tracepath and qdisc stats — if you see retransmits under 0.5% but delivery rate still low, look at the buffers, not the congestion algorithm.

Another pattern that trips up experienced teams: treating deduplication as a free win. On long-haul CDN fills, cross-stream deduplication can shave 25-30% off bandwidth for software updates or replicated container layers, but it adds 40-70µs of per-packet processing latency. At 150ms RTT that’s negligible, but if you’re also doing TLS fingerprinting and header re-write on the same appliance, you’ll saturate a single vCPU at around 4-5 Gbps. We’ve moved to a split model — light, stateless compression/FEC on the edge routers, heavy dedup only at 3-4 super-PoP aggregation points. Your CDN Backbone Topology Design: Ring, Mesh, and Hierarchical Architectures dictates where that split makes sense. In a hierarchical backbone, optimize at the aggregation layer; in a full mesh, you’ll need more distributed, lighter optimizers.

Finally, don’t ignore forward error correction tuning. Default FEC (like 10% overhead) is wasteful on clean fiber and insufficient on lossy transatlantic paths during storm season. We run adaptive FEC that scales from 2% overhead at <0.05% loss to 15% overhead at >1% loss, driven by real-time loss telemetry every 500ms. On a 180ms link with 0.8% loss, that adaptive approach recovered 18% more goodput than static FEC and avoided the retransmit storm that was adding another 250ms of tail latency to cache fills. If your fills are consistently missing your 1-second SLO, adaptive FEC is often the missing piece you haven’t instrumented yet.

Frequently Asked Questions