Bandwidth-Delay Product and CDN Tuning: Filling the Pipe Without Overwhelming It

Last year I watched a customer upgrade their origin to 100 Gbps and call us in a panic because their cache fill from Singapore to Los Angeles was still crawling at 1.8 Gbps. They’d thrown bandwidth at the problem, but the pipe was 95% empty. That’s the bandwidth-delay product trap — you can have the fattest link in the world, and if your TCP stack isn’t tuned to keep enough unacknowledged data in flight, you’ll never fill it.

The Math Behind an Empty Pipe: Why BDP Dictates Throughput

Bandwidth-delay product isn’t an abstract formula; it’s literally how many bytes you must have in flight to keep the bottleneck link busy. BDP = Bottleneck Bandwidth * Round-Trip Time. If your congestion window or receive window is smaller than that number, your throughput is capped at Window / RTT, regardless of how much capacity you’ve paid for. I learned this the hard way back in 2016 when we were tuning a new PoP in Johannesburg. On paper we had 20 Gbps of transit, but a single TCP flow to London with a 142ms RTT and a default 64KB window maxed out at 3.6 Mbps. We weren’t congested, we were starving.

That starvation happens because TCP is fundamentally ACK-clocked. You send a window, you wait. If the BDP for a 10 Gbps link with 80ms RTT is 100 MB, but your host only allows a 4 MB socket buffer, you’ve left 96% of the pipe idle. This is why TCP Window Scaling and Its Impact on CDN Throughput: Why Defaults Fall Short matters so much — without scaling (RFC 7323), you’re hard-capped at 64KB, which is useless for any long-haul CDN path. And your RTT input has to be honest. A smoothed RTT that includes queuing delay or an edge load balancer’s internal hop will inflate your BDP calculation and push you to over-buffer. We now base all our tuning on edge-measured TCP RTTs using the approach described in RTT Measurement Methodology for CDN: Getting Accurate Round-Trip Times, not synthetic pings.

Here’s the quick calculation we run for every new route before we touch any sysctl. It’s simple, but I still see teams get the units wrong — bits vs bytes will kill you:

# bdp.py - calculate BDP and window-limited throughput
# bandwidth in bits per second, rtt in seconds

def bdp_bytes(bandwidth_bps, rtt_s):
    return (bandwidth_bps * rtt_s) / 8

def max_throughput(window_bytes, rtt_s):
    return (window_bytes * 8) / rtt_s  # in bps

# Example: Cache fill - 25 Gbps * 75ms RTT (Ashburn -> Frankfurt)
bandwidth = 25_000_000_000
rtt = 0.075
bdp = bdp_bytes(bandwidth, rtt)
print(f"BDP: {bdp / 1024 / 1024:.2f} MB ({bdp:.0f} bytes)")
print(f"Needed window: {bdp / 1024:.0f} KB")

# What happens with default 1MB autotune max?
limited = max_throughput(1 * 1024 * 1024, rtt)
print(f"Throughput with 1MB window: {limited / 1e9:.2f} Gbps (link is 25 Gbps)")
# Output:
# BDP: 223.52 MB (234375000 bytes)
# Needed window: 228881 KB
# Throughput with 1MB window: 0.11 Gbps (link is 25 Gbps)

TCP Tuning for Pipe Filling: Initial Windows, Autotuning, and BBR

Knowing your BDP is step one. Tuning the stack to actually achieve it without manual window hacks is step two. For years the CDN world relied on CUBIC with aggressively raised initcwnd and socket buffers. We’d set initcwnd to 32 or 64 MSS to get that first RTT burst out faster — crucial for small objects where you never reach steady state. That helped, but it didn’t solve the sustained throughput problem for large video segments or cache fills. If autotuning is too conservative or the receive window doesn’t grow fast enough, you’ll spend half the transfer ramping up and never hit BDP.

These days I tune around three knobs together: initial window, buffer autotuning limits, and congestion control. On our Linux edge nodes (5.15+), we don’t statically set a giant tcp_rmem/tcp_wmem. We let autotuning do its work but we raise the ceiling. The defaults on most distros — tcp_rmem of 4K 87380 6M — are still far too small for intercontinental 100 Gbps paths where BDP can exceed 100 MB. We typically run with a max of 64M or 128M and let BBR handle the pacing. BBRv2/v3 is a game-changer here because it explicitly models BDP instead of blindly growing cwnd until loss. It estimates bottleneck bandwidth and RTT, then paces to keep one BDP in flight plus a small extra. You’re not guessing; you’re controlling pipe filling directly. For background on how the specs expect you to handle this, the HTTP Working Group Specifications and the practical guidance on web.dev Articles are solid references for how window and priority interactions affect real page loads.

This is what our baseline tuning looks like for a high-BDP cache edge. Don’t copy-paste this to every PoP — intra-region PoPs with <5ms RTT need much smaller buffers or you’ll just add bufferbloat:

# /etc/sysctl.d/99-high-bdp-edge.conf
# Allow window scaling and large autotuning limits
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_sack = 1

# Raise autotuning max to allow BDP > 100MB
# min default max (bytes)
net.ipv4.tcp_rmem = 4096 87380 67108864
net.ipv4.tcp_wmem = 4096 65536 67108864
net.core.rmem_max = 67108864
net.core.wmem_max = 67108864
net.core.netdev_max_backlog = 100000

# BBR with pacing - let BBR model BDP instead of loss-based growth
net.ipv4.tcp_congestion_control = bbr
net.core.default_qdisc = fq

# Faster ramp for small/medium objects
# initcwnd 32 ~ 46KB, good for 0-RTT-like first burst without overwhelming
# Set per-route via `ip route change ... initcwnd 32` not globally
# ss -i will show cwnd and rtt to verify BDP tracking

When Filling Becomes Flooding: Identifying the Overfill Cliff

There’s a cliff on the other side of BDP. Fill it exactly and you get line rate with minimal queuing. Overfill it by even 20% and you start building standing queues, inflating RTT, and triggering loss that collapses throughput. I’ve seen this every time someone sets tcp_wmem max to 256M “just to be safe” on a PoP that serves a lot of mobile users. BDP for a 20 Mbps LTE user with 90ms RTT is only 225 KB. If your edge is willing to buffer 64 MB for that flow, you aren’t filling the pipe, you’re flooding the last-mile buffer and creating 2 seconds of bufferbloat. The retransmissions and jitter then destroy your video rebuffer rate.

The trick is to monitor inflight vs BDP, not just throughput. We graph cwnd, delivery rate, and min_rtt from tcp_info on every edge, and we alert when bytes_in_flight consistently exceeds BDP * 1.25 or when RTT variance spikes. That’s also where percentile analysis saves you. An average RTT of 60ms might hide that your p95 is 180ms due to queuing on overfilled paths. If you tune for p50 you’ll underfill for most users; if you tune for p99 you’ll flood the median. Our capacity team uses CDN Percentile Latency Analysis: What p50, p95, and p99 Actually Tell You to set per-region BDP targets — typically p75 for throughput-sensitive cache fill, p90 for latency-sensitive web.

To make this concrete, here are real BDP numbers we use for capacity planning. Note how quickly the required window explodes with distance, even at modest bandwidths. If you’re still running a 6 MB default, you’re capped on three of these four paths:

From Theory to Edge: Practical BDP-Aware CDN Configuration

Knowing your BDP is one thing — configuring your edge to actually respect it in production is where most teams stumble. You can't just set a global 64MB TCP window and call it a day. A PoP in Frankfurt serving fiber users at 8ms RTT and 500 Mbps needs a ~500KB BDP, while the same PoP serving transatlantic users at 110ms and 50 Mbps needs ~687KB. If you've got one static buffer, you're either starving the high-latency pipe or bloating the low-latency one. We've learned to measure BDP per route, not per server.

Start by pulling real RTT and throughput percentiles from your edge logs, not synthetic pings. Look at your p95 RTT to each major eyeball ASN and the 90th percentile goodput you've actually achieved. That's your target BDP. Then tune the stack to let the congestion window grow that large without application-level caps. For Linux edges, that means enabling window scaling, raising max buffers, and letting autotuning do its job — but with guardrails. Don't forget, throughput vs. bandwidth aren't the same; your BDP calculation should use achievable throughput, not the NIC's 100 Gbps line rate, or you'll massively over-allocate.

For modern CDNs, initial window (IW10 vs IW32) is your biggest lever for short flows. Bumping IW from 10 to 32 MSS shaved 280ms off our Time-to-First-Byte for 50KB assets on 120ms RTT paths — that's a full round trip saved. But it's a double-edged sword. On congested last-mile links, that aggressive burst creates self-inflicted loss. We run IW32 only where we've verified low loss (<0.5%) and ECN support, and fall back to IW16 elsewhere. It's not set-and-forget; it's per-prefix policy.

# /etc/sysctl.d/99-bdp-tuning.conf - tuned for 10Gbps edge, mixed RTT 10-150ms
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 1
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_slow_start_after_idle = 0
net.ipv4.tcp_mtu_probing = 1
# For BBRv2 edges - let BBR handle cwnd, but give it room
net.ipv4.tcp_congestion_control = bbr

# nginx / ATS - don't cap what TCP can do
proxy_buffering on;
proxy_buffer_size 32k;

Advanced Patterns: BBR, Bufferbloat, and When Filling Goes Wrong

Once your windows are right, the congestion control algorithm decides how you fill that pipe. CUBIC will blindly grow its window until loss tells it to back off — which, by definition, means you've already overwhelmed a buffer somewhere. That's why we've moved 80% of our fleet to BBRv2. BBR models the BDP directly and tries to operate at the Kleinrock optimal point: exactly one BDP in flight, no more. In practice, we saw a 23% drop in p99 retransmits and 15% higher goodput on lossy 4G paths after switching, because it doesn't treat random wireless loss as congestion.

But even BBR can't save you if the path's buffers are broken. The telltale sign you've crossed from filling to flooding is bufferbloat — latency skyrockets under load while throughput flatlines. If your RTT jumps from 40ms idle to 400ms under a 10MB transfer, you're overfilling. Keep an eye on your CDN jitter measurements; a jitter increase from 3ms to 25ms+ during peak throughput is an early warning before packet loss even shows up. We've fixed this by pacing egress at the edge (fq qdisc with pacing rate = 1.25 * BDP / RTT) instead of bursting line-rate. It smooths the flow and keeps bottleneck queues at ~0.5x BDP, not 5x.

Another silent killer is packet reordering, which mimics loss and tricks the sender into halving its window prematurely. On anycasted or ECMP-heavy paths, we see up to 2-3% reordering, which can collapse CUBIC's throughput by 40% as it falsely enters fast recovery. If your retransmits are high but your loss counters on intermediate routers are clean, check for reordering with TCP's DSACK and reordering metrics. Mitigate it by increasing `tcp_reordering` to 20-30, enabling RACK, and avoiding per-packet load balancing at your L4 tier. It's not your BDP math that's wrong — it's the network lying to TCP about what loss really is.

Frequently Asked Questions