Weighted Round-Robin for CDN: Proportional Traffic Distribution Across Heterogeneous Servers

Back in 2019, we upgraded half the edge nodes in our Chicago PoP to new 32-core machines while the other half stayed on older 16-core boxes. Within an hour, the old servers were melting at 95% CPU while the new ones sat around 30%. We were running plain round-robin, so every server got the same number of requests regardless of what it could actually handle. That night taught me a lesson I've carried since: when your hardware isn't identical, equal distribution isn't fair distribution.

Why Equal Distribution Breaks on Heterogeneous CDN Hardware

Most CDN teams start with simple round-robin because it's easy to reason about and trivial to implement. Request one goes to server A, request two to B, request three to C, then back to A. I've deployed it myself for smaller, uniform clusters and it works fine - until it doesn't. The moment you introduce different server generations, different instance types in the cloud, or even just servers with varying cache heat, that neat equal split starts to hurt you. The weaker nodes get overloaded, latency spikes, and you start seeing 502s from the very servers you thought you were protecting.

In a real CDN, heterogeneity is the norm, not the exception. You might have bare-metal caches with 128GB RAM and NVMe drives sitting next to virtualized transcoders, or you might be running a mixed fleet of c6i.2xlarge and c6i.4xlarge instances because you scaled incrementally. If you're operating across multiple regions, GSLB Implementation Patterns: Global Server Load Balancing for Multi-Region CDN shows how this problem compounds at the geographic layer. Even with perfect health checks, plain round-robin doesn't know that server C can handle four times the throughput of server A. It just keeps handing out requests like dealing cards.

That's where weighted round-robin (WRR) earns its keep. Instead of treating every node equally, you assign each server a weight that reflects its actual capacity. A server with weight 4 gets roughly four times the requests of a server with weight 1 over any given window. It's still deterministic and predictable, which I love for debugging, but it's proportional. Here's the simplest way I explain the core logic to junior engineers when we're whiteboarding distribution:

# Simplified Weighted Round-Robin selector
# Weights reflect capacity: e.g., based on CPU cores, throughput tests

servers = [
    {"id": "edge-a", "weight": 1},  # older 8-core
    {"id": "edge-b", "weight": 2},  # newer 16-core
    {"id": "edge-c", "weight": 4},  # new 32-core NVMe
]

# Build weighted sequence: [a, b, b, c, c, c, c]
weighted_list = []
for s in servers:
    weighted_list.extend([s["id"]] * s["weight"])

# Smooth distribution via current_weight counter (Nginx-style)
def wrr_next(servers):
    # Sort by current_weight, pick highest, then subtract total weight
    # This avoids burstiness vs. naive expanded list
    total = sum(s["weight"] for s in servers)
    for s in servers:
        s["current_weight"] = s.get("current_weight", 0) + s["weight"]
    chosen = max(servers, key=lambda x: x["current_weight"])
    chosen["current_weight"] -= total
    return chosen["id"]

# 7 requests -> c, b, a, c, b, c, c (proportional, but smoothed)
for _ in range(7):
    print(wrr_next(servers))

How Weighted Round-Robin Achieves Proportional Distribution

The code above shows two approaches and the difference matters in production. The naive method is to expand the list by weight and just iterate through it. It works, but you'll send four requests in a row to your biggest server, which can create micro-bursts and hurt tail latency. What most modern proxies do - and what Nginx, HAProxy, and Envoy implement under the hood - is smooth weighted round-robin. They keep a running current_weight for each node, add the static weight on each round, pick the highest, and then subtract the total weight. The result is still proportional over time, but requests are interleaved more evenly.

I've found that smoothness is critical for CDN balancing where cache fill and origin fetch matter. If you slam your high-weight origin shield with back-to-back requests for uncached content, you'll trigger origin contention even though the average load looks right. Smooth WRR spreads that load out. It also interacts more cleanly with HTTP/2 and HTTP/3 connection coalescing. When you're multiplexing many streams over a single connection, as described in the HTTP Working Group Specifications, bursty assignment can stall entire connections. A smoother distribution keeps connection utilization balanced too, which the web.dev Articles on optimizing content delivery often highlight for performance.

You'll often hear WRR compared to other strategies. Least-Connections Load Balancing for CDN: Dynamic Distribution Based on Server Load is fantastic when request costs vary wildly or when you're dealing with long-lived connections like WebSockets, because it reacts to real-time load instead of static capacity. And if you need stickiness, IP Hash for CDN Session Persistence: Consistent User Routing Without Cookies is the go-to for routing the same client to the same cache without cookies. But for general-purpose HTTP traffic across heterogeneous caches where requests are relatively short and similar in cost, WRR gives you 90% of the benefit with far less state tracking. You don't need to poll active connection counts every few milliseconds; you just need accurate weights.

# Nginx-style WRR upstream - weights tuned from load tests
# This is from a real edge PoP with mixed instances

upstream cdn_edge {
    # Older generation - lower weight, will be drained
    server 10.0.1.11:8080 weight=2 max_fails=3 fail_timeout=30s;
    server 10.0.1.12:8080 weight=2 max_fails=3 fail_timeout=30s;

    # Current standard - 2x capacity
    server 10.0.1.21:8080 weight=4 max_fails=3 fail_timeout=30s;
    server 10.0.1.22:8080 weight=4 max_fails=3 fail_timeout=30s;
    server 10.0.1.23:8080 weight=4 max_fails=3 fail_timeout=30s;

    # New high-density nodes - 4x capacity, NVMe cache
    server 10.0.1.31:8080 weight=8 max_fails=3 fail_timeout=30s;
    server 10.0.1.32:8080 weight=8 max_fails=3 fail_timeout=30s;

    keepalive 64;
}

server {
    listen 443 ssl http2;
    location / {
        proxy_pass http://cdn_edge;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
    }
}

Calibrating Weights From Real Capacity, Not Just Spec Sheets

Setting weights by CPU core count alone is a trap I've fallen into. On paper, a 32-core box should be twice as capable as a 16-core one, but if your workload is cache-bound, memory bandwidth and SSD IOPS matter more than cores. If you're doing TLS termination or image optimization at the edge, then CPU and even crypto acceleration start to dominate. I don't set weights until we've run synthetic and real traffic replay tests at the PoP.

Our process is pretty consistent now. We isolate a single server of each tier, warm its cache for an hour under production traffic mirrored at low volume, then ramp synthetic GETs until p95 latency crosses our 120ms threshold or CPU hits 75% sustained. That requests-per-second number becomes our baseline capacity. We then normalize those numbers into small integer weights. You don't want weights like 87 and 143; keep them as 2, 4, 8 or 3, 6, 9 so the distribution math stays clean and debuggable. Re-tune monthly, or whenever you change your cache software, kernel, or NIC drivers - I've seen a kernel upgrade shift capacity by 15% with no hardware change at all.

The table below is from that Chicago PoP I mentioned, after we actually measured instead of guessed. We stopped using plain round-robin that same weekend and the difference was immediate. No more hot nodes, origin fetches dropped because the bigger caches were finally getting proportional fill, and our p99 latency flattened out.

Putting Weights Into Production: Nginx, HAProxy, and Custom Edge Logic

You've got calibrated weights — now you've got to make your edge actually honor them. And this is where a lot of teams stumble. They've done the hard work of benchmarking, derived a clean ratio like 5:3:2, and then drop it into a load balancer that uses a naive implementation that clumps traffic. You'll see bursts of 5 requests hammer the big box, then 3 hit the medium one, then nothing for a second. On heterogeneous hardware, that burstiness kills your tail latency.

Most teams start with what they've got. In Nginx, it's dead simple and surprisingly effective for 90% of CDN workloads. We've run this in production fronting origins with a 40 Gbps aggregate. The weight parameter isn't just a preference — Nginx's smooth weighted round-robin ensures the distribution is interleaved, not batched. Don't set them once and forget them, though. Pair weights with active failure detection so a slow origin doesn't keep getting its full share.

# /etc/nginx/conf.d/cdn-origin.conf - Smooth Weighted Round-Robin
upstream cdn_origins {
    # Calibrated weights: based on p95 sustained RPS, not core count
    # srv-a: 32-core EPYC + NVMe = ~12k RPS -> weight 6
    # srv-b: 16-core Xeon + SSD  = ~6k RPS  -> weight 3
    # srv-c: 8-core legacy       = ~2.5k RPS -> weight 1

    server 10.0.1.11:8080 weight=6 max_fails=3 fail_timeout=30s slow_start=60s;
    server 10.0.1.12:8080 weight=3 max_fails=3 fail_timeout=30s slow_start=60s;
    server 10.0.1.13:8080 weight=1 max_fails=3 fail_timeout=30s slow_start=30s;

    keepalive 256;
}

server {
    location / {
        proxy_pass http://cdn_origins;
        proxy_next_upstream error timeout http_502 http_503 non_idempotent;
        proxy_connect_timeout 250ms;
    }
}

If you're past Nginx or HAProxy and building your own L7 edge in Go, Rust, or Lua, implement the smooth WRR algorithm directly. It tracks a current_weight per server, adds its configured weight each round, picks the highest, then subtracts the total weight sum. It's O(n) per pick, which is trivial when n is 20 origins, and it gives you perfect proportional interleaving. That's what prevents that 500ms cache-fill spike we saw when all large objects landed on the same high-capacity node simultaneously.

When Distribution Drifts: Dynamic Rebalancing, Health Checks, and Hot Spots

Static weights rot. We learned this the hard way during a Black Friday event where our weight=6 box was handling 58% of traffic at 9am, but by 2pm it was throttling at 85% CPU because its NVMe cache had filled and it started hitting disk. The spec sheet didn't change, but effective capacity did. You can't rely on a one-time calibration. Your system needs dynamic feedback. We now adjust weights every 30-45 seconds based on an exponentially weighted moving average of CPU load, active connections, and p95 origin latency. If a server's latency creeps past 180ms or its 5-minute load average exceeds 80%, we shave 20-30% off its weight and bleed that traffic to healthier nodes. It'll recover on its own without manual intervention.

This is also where your health checks make or break you. A naive shallow TCP check will keep sending traffic to a server that's returning 90% cache misses due to a failed SSD. You need layered checks. For a deeper dive on this, see our breakdown of Health Check Design Patterns for CDN: Shallow, Deep, and Synthetic Checks and how we combine them with Active vs Passive Health Checks for CDN: Probing vs Observation-Based Monitoring. In practice, we run a passive check (monitoring real request error rates and TTFB) that can veto the WRR decision in under 2 seconds. If passive monitoring flags an origin, we pull it from the pool regardless of its weight — no point proportionally distributing traffic to a node that's failing 15% of requests.

Finally, don't confuse weighted round-robin with consistent hashing. WRR is perfect for stateless origin pools or shield PoPs where any server can serve any object. But if you've got a large, hot cache tier where locality matters, WRR will destroy your hit ratio by spraying the same object across all nodes. In that tier, you want a Consistent Hash Ring Architecture for CDN: Minimizing Redistribution on Server Changes and then apply weights *within* the ring's virtual nodes. We've seen teams run WRR at L4 for PoP selection and consistent hashing at L7 for cache affinity — that hybrid gets you both proportional load and a 94%+ hit ratio, instead of choosing one.

Frequently Asked Questions