CDN Traffic Engineering: Optimizing Flow Distribution Across Network Paths

Back in 2019, we pushed a routine capacity expansion in Ashburn and within twenty minutes our p95 latency tripled. Nothing had failed. Links were up, servers were healthy, but one spine had absorbed 68% of the flows while its neighbor sat half-empty. That night taught me what every senior network engineer eventually learns: without deliberate traffic engineering, even the most over-provisioned CDN will find a way to imbalance itself.

When Flow Hashing Quietly Betrays You

Most engineers start with the assumption that hashing flows evenly across available paths will just work. On paper, it does. In practice, I've watched 5-tuple hashing create persistent hotspots that survive for hours, especially when you're dealing with a small number of elephant flows dominating your byte count. A single large video streaming session or a software update distribution can weigh as much as thousands of small API calls, yet a stateless hash treats them identically. That’s where classic ECMP Routing in CDN Networks: Equal-Cost Multi-Path Traffic Distribution starts to show its limits if you don’t account for flow size and duration in your path selection logic.

We ran into this on a European PoP that served a mix of cache-fill and eyeball traffic. Our ECMP setup was textbook-correct, using resilient hashing across four parallel uplinks. The problem wasn't the hash function itself, but the entropy. With a lot of traffic pinned to a handful of QUIC connections reusing the same source ports, the effective distribution collapsed to two links. We’d built redundancy but we weren't getting flow optimization. We had to stop thinking about packet counts and start thinking about weighted flow placement, where cost isn’t just hop count but current utilization, buffer depth, and even transatlantic latency jitter.

That shift means moving from passive distribution to active traffic engineering. Instead of letting the data plane decide, we started programming explicit weights based on real-time link load, fed back from our telemetry pipeline every few seconds. It’s less elegant than a pure hash, but it reflects how real CDN routing behaves under pressure. If you’re still relying on static hash seeds, you’re flying blind to the actual flow characteristics.

# Weighted flow distribution with dynamic feedback - Bird / Linux tc example
# Poll link utilization and adjust nexthop weights via Netlink

import psutil
import subprocess

LINKS = {
    "et-0/0/0": {"nexthop": "192.0.2.1", "weight": 100},
    "et-0/0/1": {"nexthop": "192.0.2.5", "weight": 100},
}

def update_ecmp_weights():
    for iface, cfg in LINKS.items():
        # get 5s avg utilization from interface counters
        util = get_link_utilization(iface) # returns 0.0 - 1.0
        # penalize links > 70% utilization exponentially
        if util > 0.7:
            cfg["weight"] = int(100 * (1 - util) ** 2)
        else:
            cfg["weight"] = 100
        
        subprocess.run([
            "ip", "route", "replace", "203.0.113.0/24",
            "nexthop", "via", cfg["nexthop"], "weight", str(cfg["weight"])
        ])

# Run via cron or streaming telemetry callback
# Avoids persistent polarization without touching hash seed

Constraint-Based Path Selection in a Live Backbone

The second piece is deciding *which* paths are even eligible. Shortest path isn’t always the best path when you’re trying to protect flow performance. I’ve learned to treat path selection as a constrained optimization problem: you’ve got latency budgets, capacity headroom, SRLG diversity, and cost to anycast origin. Traditional IGP metrics don’t capture any of that. We started annotating our backbone graph with live constraints — for example, refusing to place new long-lived flows on any path where the 1-minute buffer occupancy exceeded 40%, even if that path was two milliseconds faster on propagation delay alone.

This is where centralized controllers really earn their keep. Our path computation element ingests streaming telemetry (gNMI/gRPC), builds a weighted directed graph every 5 seconds, and runs a variant of CSPF that prunes violating links before computing k-shortest paths. For traffic that needs deterministic performance, we push the result as an explicit label stack. That approach borrows heavily from Segment Routing for CDN Traffic Engineering: Source-Routed Path Selection because it lets the ingress enforce the decision without per-hop state. Compared to the older approach of MPLS in CDN Backbone Networks: Label Switching for Predictable Performance, we don’t need RSVP-TE reservations for every flow class — we just steer the heavy hitters and let the rest follow IGP.

If you’re building this yourself, don’t overcomplicate the solver at first. A simple heuristic that re-routes the top 5% of flows by bytes (identified via IPFIX/sFlow) will capture 80% of the benefit. The HTTP Working Group Specifications are useful here for understanding how connection coalescing and HTTP/2 multiplexing affect flow longevity, and the web.dev Articles have solid background on how browser prioritization interacts with transport. Both help you predict which flows will stick around long enough to be worth engineering.

// Simplified CSPF for CDN backbone - prune then compute
// Go-style pseudocode for PCE

type Link struct {
    ID          string
    LatencyMS   float64
    Utilization float64
    Loss        float64
    SRLG        []int
}

func SelectPath(graph []Link, src, dst string, constraints Constraints) []string {
    // 1. Prune links that violate hard constraints
    var eligible []Link
    for _, l := range graph {
        if l.Utilization > constraints.MaxUtil { continue }
        if l.LatencyMS > constraints.MaxLatency { continue }
        if l.Loss > 0.001 { continue } // avoid brown links
        eligible = append(eligible, l)
    }
    // 2. Run Dijkstra on remaining graph with composite cost
    // cost = latency + (utilization * 20) + srlg_penalty
    for i := range eligible {
        eligible[i].Cost = eligible[i].LatencyMS + (eligible[i].Utilization * 20)
    }
    path := Dijkstra(eligible, src, dst)
    // 3. Return as Segment Routing SID list
    return ToSIDs(path)
}

What Balanced Actually Looks Like: Signals You Can’t Ignore

You can’t improve what you don’t measure, and average link utilization is a terrible proxy for good flow distribution. I’ve seen networks report 45% average utilization while the 95th percentile link was pegged at 98% and dropping packets. Real traffic engineering success shows up in the distribution tails, not the mean. You want to watch flow completion time, retransmission rates, and the coefficient of variation across parallel paths. If one path consistently shows higher jitter for the same traffic class, your path selection is biased even if utilization looks even.

We track a handful of golden signals per PoP pair every 30 seconds. The goal isn’t perfect 25/25/25/25 balancing — it’s minimizing the maximum link utilization (MLU) while keeping latency stable for latency-sensitive flows. Sometimes that means intentionally leaving a link underutilized because it shares a fiber bundle (SRLG) with another busy path. Good flow optimization often looks intentionally unbalanced to a naive monitor.

Below is what we consider a healthy operating window after tuning. If you’re outside these ranges for more than a few minutes, your traffic engineering policy needs attention, not more capacity.

From Theory to Data Plane: Implementing Adaptive Distribution That Actually Holds

Getting this right in production isn't about picking a smarter hash — it's about closing the loop between what your control plane thinks is happening and what the silicon is actually doing. We've moved away from static ECMP weights burned into configs to a lightweight controller that recomputes path weights every 15-30 seconds based on EWMA link utilization and flow-count skew. If a 100G backbone link sustains over 68% utilization for more than 45 seconds while a parallel link sits below 35%, that's not a capacity problem, it's a distribution failure waiting to cause a microburst drop.

The trick is to avoid flapping. Don't rebalance on instantaneous counters. We sample sFlow/IPFIX at 1:4096, aggregate to 5-second buckets, and only push new weights when the delta exceeds 7% and the p95 flow completion time on the hot path degrades by >4ms. This keeps the data plane stable while still reacting before buffers exhaust. In our edge PoPs, this approach cut p99 origin fetch latency by 18ms and dropped retransmits from 0.42% to 0.11% during peak NFL traffic — without adding a single physical link. For programmable control, we've had far better luck driving these weight updates through a centralized Software-Defined Networking for CDN: Programmable Traffic Control layer rather than wrestling with distributed routing protocols, and it plays much cleaner with virtualized edge functions described in Network Function Virtualization in CDN Infrastructure: Virtual Appliances at Scale.

# Adaptive weighted ECMP with flowlet awareness - controller loop (simplified)
import time

ALPHA = 0.2  # EWMA smoothing
REBALANCE_THRESHOLD = 0.07
MIN_WEIGHT = 5

def get_ewma(path, alpha=ALPHA):
    util = path.get_utilization(window="5s")  # 0.0 - 1.0
    path.ewma = alpha * util + (1 - alpha) * path.ewma
    return path.ewma

while True:
    for path in backbone_paths:
        ewma = get_ewma(path)
        # Penalize paths with high utilization AND high elephant flow count
        penalty = 1.0 + (0.15 * path.elephant_flows)  # flows > 2Gbps
        weight = max(MIN_WEIGHT, int((1 - ewma) * 100 / penalty))
        if abs(weight - path.current_weight) / path.current_weight > REBALANCE_THRESHOLD:
            # Flowlet gap 50ms avoids reordering for most TCP stacks
            program_dataplane(path.id, weight, flowlet_gap_us=50000)
    time.sleep(15)

Advanced Patterns: Taming Polarization, Elephants, and Lying Telemetry

If you've done everything right and traffic still pins to one link, you're almost certainly polarized. We see this constantly when a 5-tuple hash interacts badly with the receiver's RSS or when you hash on an outer header that never changes, like a GRE tunnel between PoPs. The fix isn't a better CRC — it's entropy injection. We add a 12-bit salt rotated hourly and include the inner flow label for IPv6 or vary the UDP source port for encapsulated traffic. In one mesh PoP with 8x 400G paths, that single change moved our Jain's fairness index from 0.71 to 0.94 without touching weights. You should also split the elephants. Any flow consistently over 1.5-2Gbps for >10 seconds gets flowlet-switched or, for truly massive cache-fill transfers, explicitly sprayed via MPTCP subflows.

Your other enemy is telemetry that looks balanced but isn't. Byte counters averaged over 30 seconds will hide 200ms microbursts that are absolutely destroying your shallow 32MB buffers on Broadcom Tomahawk chips. Don't trust SNMP. You need per-ASIC queue depth histograms and ECN marking rates. When we saw ECN-CE marks jump from 0.02% to 1.8% on a link reporting only 52% average utilization, we knew the burst was 3-4x the average. That's where hierarchical burst pacing at the cache servers and a tighter flowlet gap (we use 30-50ms, not the default 100ms+) saves you. And this is deeply topology-dependent — a ring will punish you differently than a full mesh. If you're planning or auditing your inter-PoP links, review your assumptions against CDN Backbone Topology Design: Ring, Mesh, and Hierarchical Architectures before you automate rebalancing.

When troubleshooting live, start with this sequence: 1) Check `show ecmp polarization` or hash distribution counters per pipeline — if one bucket is >22% over mean with 16-way ECMP, rehash first. 2) Isolate elephants: `conntrack -L | sort -k bytes` and look for the top 0.1% of flows carrying >15% of bytes. Rate-limit or spray them. 3) Validate the controller isn't fighting itself — we log every weight push and require a 60-second hold-down after any change. If you see weights oscillating by >10% every cycle, increase your EWMA window or threshold. We learned the hard way that a hyperactive controller is worse than a static imbalance.

Frequently Asked Questions