TCP BBR Congestion Control: How CDN Edge Servers Optimize Transport

Every millisecond of transport-layer inefficiency multiplies across millions of concurrent CDN connections. Traditional loss-based congestion control algorithms like CUBIC and Reno were designed for wired networks where packet loss reliably signals congestion. On modern networks — where wireless interference, shallow buffers, and path diversity cause non-congestion losses — these algorithms chronically underutilize available bandwidth. TCP BBR (Bottleneck Bandwidth and Round-trip propagation time) takes a fundamentally different approach by building a model of the network path and sending at the mathematically optimal rate. For CDN edge servers handling connections across every network type on earth, BBR delivers measurably better throughput, lower latency, and more consistent performance. This guide covers BBR's internals, compares it against loss-based alternatives, and walks through production deployment on CDN edge servers.

The Problem with Loss-Based Congestion Control

Loss-based algorithms share a common model: increase the sending rate until packets are dropped, interpret the drop as congestion, reduce the rate, and repeat. CUBIC, the default on Linux since 2.6.19, uses a cubic function to grow the congestion window after a loss event, reaching the previous window size quickly and then probing beyond it slowly. Reno uses a simpler additive-increase/multiplicative-decrease (AIMD) pattern. Both algorithms use packet loss as their primary signal.

This model breaks down on modern networks for three reasons. First, non-congestion packet loss is common: WiFi retransmissions, LTE handoffs, and random bit errors all drop packets without indicating the path is overloaded. When CUBIC sees a 1% random loss rate on a mobile network, it halves its window repeatedly, achieving perhaps 10% of the link's actual capacity. Second, bufferbloat — the tendency of network devices to have oversized buffers — means loss-based algorithms fill queues to overflowing before detecting congestion. The result is high latency (hundreds of milliseconds of queuing delay) with the algorithm unable to distinguish its own queue-building from network congestion. Third, in CDN environments where edge servers maintain thousands of simultaneous connections, the aggregate effect of every connection probing for bandwidth by inducing loss creates a noisy, suboptimal equilibrium.

How BBR Models the Network Path

BBR's core insight is that the optimal sending rate for any network path can be determined from two measurements: the bottleneck bandwidth (BtlBw) — the maximum rate the path can deliver data — and the round-trip propagation time (RTprop) — the minimum latency of an unloaded path. The product of these two values gives the bandwidth-delay product (BDP), which is the exact amount of data that should be "in flight" at any time. Sending less than the BDP underutilizes the link. Sending more fills router queues, adding latency without increasing throughput.

The challenge is that BtlBw and RTprop cannot be measured simultaneously. Measuring BtlBw requires sending enough data to saturate the bottleneck, which fills queues and inflates RTT. Measuring RTprop requires draining queues, which means sending below the bottleneck rate. BBR solves this by measuring each parameter in alternating phases.

The Four Phases of BBR

Startup: When a connection begins, BBR doubles its sending rate each round trip (similar to TCP slow start) until the delivery rate stops increasing. This signals that the bottleneck bandwidth has been reached. BBR records this as the initial BtlBw estimate. Startup typically completes in log2(BDP/MSS) round trips.

Drain: During Startup, BBR overshoot the optimal rate to find BtlBw, building a queue at the bottleneck. The Drain phase sends below the estimated BtlBw to empty this queue. Drain ends when the estimated inflight data equals the BDP (BtlBw × RTprop).

ProbeBW: This is BBR's steady state, where it spends the vast majority of the connection's lifetime. BBR cycles through eight phases of pacing gain: one at 1.25× (probing for higher bandwidth), one at 0.75× (draining any queue built), and six at 1.0× (cruising at the estimated optimal rate). Each phase lasts for the estimated RTprop duration. This cycle lets BBR detect bandwidth increases within 8 RTTs while keeping queue occupancy low.

ProbeRTT: Every 10 seconds (in BBRv1), BBR reduces its inflight data to just 4 packets for 200 milliseconds to get an uncontaminated RTprop measurement. This is necessary because as the algorithm sends data, it inevitably adds some queuing delay, and RTprop would drift upward without periodic probing. This phase causes a brief throughput dip but ensures the model stays accurate.

# BBR state machine pseudocode
class BBRStateMachine:
    def __init__(self):
        self.state = "Startup"
        self.btl_bw = 0          # Max observed delivery rate
        self.rt_prop = float('inf')  # Min observed RTT
        self.pacing_gain = 2.0   # Startup doubles each RTT

    def on_ack(self, sample):
        # Update BtlBw (windowed max over 10 RTTs)
        self.btl_bw = max(self.btl_bw, sample.delivery_rate)
        # Update RTprop (windowed min over 10 seconds)
        self.rt_prop = min(self.rt_prop, sample.rtt)
        # Calculate optimal sending rate
        bdp = self.btl_bw * self.rt_prop
        self.pacing_rate = self.btl_bw * self.pacing_gain
        self.cwnd = bdp * self.cwnd_gain

    def transition(self):
        if self.state == "Startup":
            if not self.bandwidth_growing():
                self.state = "Drain"
                self.pacing_gain = 0.75
        elif self.state == "Drain":
            if self.inflight <= self.bdp():
                self.state = "ProbeBW"
        elif self.state == "ProbeBW":
            if self.time_to_probe_rtt():
                self.state = "ProbeRTT"
        elif self.state == "ProbeRTT":
            if self.probe_rtt_done():
                self.state = "ProbeBW"

BBR vs CUBIC: Performance Comparison for CDN Workloads

The performance gap between BBR and CUBIC depends heavily on the network conditions typical of CDN delivery. On low-latency, low-loss wired connections, both algorithms perform similarly because CUBIC's loss signals are reliable. The divergence appears on the networks where CDN performance matters most.

High-loss mobile networks (1-5% random loss): CUBIC's throughput drops nearly proportionally with loss rate because each loss triggers a multiplicative decrease. At 2% random loss, CUBIC typically achieves 30-40% of the link's capacity. BBR maintains 90%+ utilization because its bandwidth estimate is based on delivery rate observations, not loss absence. Google reported that switching YouTube to BBR increased throughput by 4% globally and up to 14% in developing regions with lossy last-mile infrastructure.

Long-distance connections (high BDP): A 100 Mbps link with 200ms RTT has a BDP of 2.5 MB. CUBIC needs to grow its window to 2.5 MB to fully utilize this link, then drops to 1.25 MB on any loss event. The recovery back to 2.5 MB takes significant time due to CUBIC's growth function. BBR immediately sends at the measured BtlBw regardless of BDP size, achieving full utilization within the first few round trips after startup.

Bufferbloated paths: CUBIC fills available buffers before detecting loss, adding hundreds of milliseconds of queuing delay. BBR explicitly avoids filling buffers by pacing at the bottleneck bandwidth rather than probing for loss. On bufferbloated paths, BBR delivers similar throughput to CUBIC but with dramatically lower latency — often reducing RTT from 500ms+ to under 50ms.

Deploying BBR on CDN Edge Servers

Enabling BBR on a Linux CDN edge server requires kernel 4.9 or later (BBRv2 requires 5.18+) and two sysctl settings. The critical requirement is pairing BBR with the fq (Fair Queuing) qdisc, which implements packet pacing. Without fq, BBR sends data in bursts rather than evenly-spaced packets, which reduces its effectiveness on shared links.

# Enable BBR with fq pacing
echo "net.core.default_qdisc = fq" >> /etc/sysctl.d/90-bbr.conf
echo "net.ipv4.tcp_congestion_control = bbr" >> /etc/sysctl.d/90-bbr.conf
sysctl -p /etc/sysctl.d/90-bbr.conf

# Verify BBR is active
sysctl net.ipv4.tcp_congestion_control
# Output: net.ipv4.tcp_congestion_control = bbr

# Verify fq qdisc is default
sysctl net.core.default_qdisc
# Output: net.core.default_qdisc = fq

# Check available congestion control algorithms
sysctl net.ipv4.tcp_available_congestion_control
# Output: reno cubic bbr

Kernel Tuning for BBR at Scale

CDN edge servers handling tens of thousands of concurrent connections need additional kernel tuning beyond enabling BBR. The default Linux TCP buffer sizes are too small for high-BDP paths, and the socket memory limits can restrict BBR's ability to maintain optimal inflight data.

# TCP buffer tuning for CDN edge servers
# Min, default, max buffer sizes in bytes
net.ipv4.tcp_rmem = 4096 131072 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216

# Allow BBR to use larger windows for high-BDP paths
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 1

# Socket memory pages (total system limit)
net.ipv4.tcp_mem = 786432 1048576 1572864

# Enable TCP Fast Open for reduced connection latency
net.ipv4.tcp_fastopen = 3

# Reduce TIME_WAIT sockets for high-connection-rate servers
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15

For protocol optimization beyond congestion control, consider enabling TCP Fast Open (TFO) alongside BBR. TFO eliminates one round trip on repeat connections by including data in the SYN packet, complementing BBR's transport efficiency with reduced connection setup latency.

BBRv2: Addressing BBRv1's Limitations

BBRv1 has known issues that BBRv2 addresses. The most significant is fairness: when BBRv1 flows share a bottleneck with CUBIC flows, BBRv1 tends to capture more than its fair share of bandwidth because it does not reduce its sending rate in response to loss (which is the mechanism CUBIC uses to share capacity). In some cases, a single BBRv1 flow could take 40%+ of a shared link while dozens of CUBIC flows share the remainder.

BBRv2 introduces several improvements. It uses ECN (Explicit Congestion Notification) as a congestion signal, responding to ECN marks by reducing the inflight target. This allows BBRv2 to detect congestion before loss occurs and share bandwidth more equitably with ECN-aware loss-based flows. BBRv2 also adds an explicit loss tolerance parameter: instead of ignoring all loss as BBRv1 does, BBRv2 treats loss rates above a configurable threshold (default 2%) as a congestion signal. The ProbeRTT phase is redesigned to be less disruptive — instead of reducing to 4 packets for 200ms, BBRv2 uses a shorter, more frequent probing strategy that causes smaller throughput dips.

# Check BBR version (Linux 5.18+)
cat /proc/sys/net/ipv4/tcp_congestion_control
# bbr    (BBRv1 on kernels 4.9-5.17)
# bbr2   (BBRv2 on kernels 5.18+, if compiled)

# On BBRv2-capable kernels, both versions may be available
cat /proc/sys/net/ipv4/tcp_available_congestion_control
# reno cubic bbr bbr2

# Switch to BBRv2
sysctl -w net.ipv4.tcp_congestion_control=bbr2

# BBRv2 tunable parameters (advanced)
# Loss tolerance threshold (percent, default 2)
echo 2 > /proc/sys/net/ipv4/tcp_bbr2_loss_thresh
# ECN response aggressiveness
echo 1 > /proc/sys/net/ipv4/tcp_bbr2_ecn_enable

Monitoring BBR Performance in Production

Effective BBR deployment requires monitoring to validate performance gains and detect issues. The key metrics to track are delivery rate, RTT distribution, retransmission rate, and congestion window size. Linux's ss utility provides per-connection BBR state information that is invaluable for debugging.

# View BBR state for active connections
ss -tin | grep bbr
# Shows: bbr wscale:7,7 rto:204 rtt:23.5/1.2 ato:40
#        mss:1448 pmtu:1500 rcvmss:1448 advmss:1448
#        cwnd:42 ssthresh:34 bytes_sent:1584000
#        bytes_retrans:0 bytes_acked:1584000
#        delivery_rate 52.3Mbps

# Parse BBR-specific metrics
ss -tin | awk '/bbr/{
    match($0, /delivery_rate ([0-9.]+)/, dr)
    match($0, /rtt:([0-9.]+)/, rtt)
    match($0, /cwnd:([0-9]+)/, cw)
    match($0, /bytes_retrans:([0-9]+)/, ret)
    printf "Rate: %s  RTT: %sms  CWND: %s  Retrans: %s\n",
        dr[1], rtt[1], cw[1], ret[1]
}'

# Aggregate BBR stats with SystemTap or eBPF
# /usr/share/bcc/tools/tcpretrans  (retransmission events)
# /usr/share/bcc/tools/tcplife     (connection lifetime stats)

For CDN-wide monitoring, export per-connection BBR metrics to your observability stack. Key aggregations include: P50/P95/P99 delivery rate by client ASN and geographic region, percentage of connections spending significant time in ProbeRTT (indicating potential issues), retransmission rate distribution (BBRv1 should be under 3%, BBRv2 under 1%), and RTT inflation compared to baseline (detecting bufferbloat regression).

BBR Deployment Considerations and Trade-offs

BBR is not universally superior. In controlled environments with reliable loss signals (datacenter networks, for example), CUBIC performs comparably and has decades of deployment experience. BBRv1's fairness issues mean deploying it on shared infrastructure can harm other tenants. And BBR's bandwidth probing can trigger rate limiters or traffic shapers that interpret the probing phase as suspicious traffic bursts.

For CDN edge servers, the recommendation is clear: deploy BBRv2 where kernel support is available (5.18+), BBRv1 on older kernels with awareness of its fairness limitations, and keep CUBIC as the fallback. Use per-connection traffic management policies to select the congestion control algorithm based on connection characteristics: BBR for long-haul and mobile connections where its model-based approach shines, CUBIC for local datacenter or intra-PoP connections where loss signals are reliable.

The combination of BBR for transport efficiency, latency optimization at the CDN layer, and proper kernel tuning creates a transport stack that extracts maximum performance from every network path. For CDN operators serving global audiences across heterogeneous networks, BBR is not just an optimization — it is a fundamental shift in how edge servers communicate with the internet.