Least-Connections Load Balancing for CDN: Dynamic Distribution Based on Server Load

Back in 2019, we had a major streaming customer push a live finale to just under two million concurrent viewers. Our edge tier was running classic round-robin and looked perfectly balanced on paper, yet two nodes in Ashburn were melting while three others sat at 40% CPU. The problem wasn't traffic volume — it was connection duration. Long-lived HLS sessions stuck to the overloaded machines for minutes, and the balancer kept blindly spraying new users evenly. That's the day we stopped trusting static distribution for anything with persistent connections.

Why Fixed Ratios Collapse When Connections Live Longer

Weighted round-robin is a solid starting point, and I've used it for years on heterogeneous hardware where you've got a mix of older and newer edge servers. You assign a weight, the balancer hands out requests proportionally, and for short, uniform requests it works. We cover that approach in detail in Weighted Round-Robin for CDN: Proportional Traffic Distribution Across Heterogeneous Servers, and it’s still my default when I'm dealing with small objects and predictable CPU cost. But it doesn't see what's actually happening right now.

The issue is that not all connections cost the same. A 2KB API check and a 4K video segment request might both count as "one request," but one holds a socket open for 80ms and the other for 45 seconds with TLS, TCP congestion control, and origin revalidation tangled in. If your balancer only counts requests handed out, not requests still active, you'll create hot spots without realizing it. I saw this repeatedly with image-heavy e-commerce sites during flash sales: the balancer thought it was being fair, while our p95 latency on two nodes spiked to 1.8s because they were juggling 12k active connections each versus 4k on their neighbors.

Connection-aware balancing flips that logic. Instead of asking "who's turn is it?" it asks "who's least busy right now?" The load balancer tracks active, in-flight connections per node in real time and sends the next request to the node with the fewest. It's simple in theory, but that real-time server load awareness is what saves you when traffic isn't uniform — which, for a CDN, it never is.

# NGINX - least connections at the edge
# This replaced our weighted round-robin for the video PoP

upstream video_edge {
    least_conn;
    server edge-a.ash1.example.net weight=3 max_fails=2 fail_timeout=10s;
    server edge-b.ash1.example.net weight=3 max_fails=2 fail_timeout=10s;
    server edge-c.ash1.example.net weight=2 max_fails=2 fail_timeout=10s;
    # slow-start helps new or recovered nodes avoid a thundering herd
    server edge-d.ash1.example.net weight=3 slow_start=30s;
    keepalive 1024;
}

server {
    listen 443 ssl http2;
    location /hls/ {
        proxy_pass http://video_edge;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
    }
}

How Dynamic Distribution Actually Picks the Next Server

At its core, least connections is just a sorted list that updates on every connect and disconnect event. Most proxies — NGINX, HAProxy, Envoy, even cloud LBs — maintain an atomic counter per upstream. When a new client request comes in, the balancer scans the counters, picks the lowest, increments it, and proxies the connection. When that connection closes or completes, it decrements. There's no hashing or weight math unless you add it. For pure dynamic distribution, the decision is made in microseconds with O(n) or O(log n) lookup depending on implementation.

Where it gets interesting is what you count as a "connection." Are you counting TCP sockets, HTTP requests over keep-alive, or active streams for HTTP/2 and HTTP/3? I've been burned by this distinction. Early on we counted TCP connections only, so an HTTP/2 edge node with 500 sockets but 8,000 multiplexed streams looked idle compared to an HTTP/1.1 node with 1,200 sockets. Now we tune for active requests. According to the HTTP Working Group Specifications, multiplexing fundamentally changes concurrency, and your balancer needs to know that. Envoy handles it better by tracking active requests rather than raw sockets, and the guidance on web.dev Articles about connection coalescing and prioritization helped us refine that counting. If you don't account for multiplexing, connection-aware balancing will make the wrong choice every time.

You also have to decide what happens with ties and flaps. If three servers all sit at 4,210 connections, you don't want to hammer the same one. Good implementations randomize among equals or fall back to round-robin for the tie-break. And you need fast failure detection. If a node stops decrementing because it's hanging, its count will look low forever. That's why we pair least connections with active health checks and a max-connections circuit breaker. It's not a replacement for GSLB Implementation Patterns: Global Server Load Balancing for Multi-Region CDN at the DNS layer, but inside a single PoP, it's the most responsive way to shed load without manual intervention.

# HAProxy - least connections with active request tracking
# and tie-break randomization

backend cdn_edge_dynamic
    balance leastconn
    option httpchk GET /healthz
    http-check expect status 200
    default-server inter 2s rise 2 fall 3 maxconn 20000 slowstart 30s
    server edge-01 10.0.1.11:443 check weight 100 track-active-requests
    server edge-02 10.0.1.12:443 check weight 100 track-active-requests
    server edge-03 10.0.1.13:443 check weight 75  track-active-requests
    # hash-type consistent  # <-- don't use this here; it defeats leastconn

# Envoy equivalent (active_request Biased)
# envoy.filters.http.least_request picks 2 random hosts,
# then chooses the one with fewer active requests

Weighted Least Connections: Making Heterogeneous Hardware Play Nice

Pure least connections assumes every server can handle the same load. In real CDN PoPs, they can't. You'll have last year's 32-core machines next to new 64-core boxes with twice the NIC throughput, or a PoP where one rack has NVMe cache and another is still on SSD. If you run unweighted, the older box will still receive the same share until it starts struggling — by then you've already hurt tail latency. I've learned to never deploy pure least connections without weights if the hardware isn't identical.

Weighted least connections solves this by dividing active connections by weight. So a server with 10,000 active connections and weight 100 has an effective load of 100, while a server with 7,000 connections and weight 50 has an effective load of 140 — the balancer will pick the first one despite its higher absolute count. The math is cheap but the result is far closer to actual server load. We apply this per PoP and let our global layer handle cross-region spillover. It’s also where you have to think about persistence trade-offs. If you need sticky sessions, least connections alone won't help you — you'll want to review IP Hash for CDN Session Persistence: Consistent User Routing Without Cookies before mixing the two, because persistence and least-busy routing pull in opposite directions.

Don't guess at the weights. Start from capacity testing, then let observed metrics drive adjustments. We run synthetic load tests with realistic object size mixes and tune weights until CPU and active connection distribution converge. Below is what that convergence typically looks like when we migrated an image and video PoP from round-robin to weighted least connections. The numbers aren't lab-perfect; they're averaged from a week of production traffic after warming the cache.

Putting Least-Connections Into Production: Edge Implementation Details

Knowing the algorithm’s great — actually running it at the edge without shooting yourself in the foot is another story. Most teams I’ve worked with start with Nginx or Envoy, and both make least-connections deceptively easy to turn on. But the defaults will bite you if you don’t tune connection tracking and health state. You can’t just count TCP sockets; you’ve got to decide what a “connection” actually means for your workload. Is it an active HTTP/2 stream? A WebSocket that’ll live for 40 minutes? A gRPC stream holding a cache-fill?

For modern CDNs, you’ll want to track active requests, not just TCP connections. Envoy’s least_request does this correctly out of the box, while Nginx’s least_conn counts active connections by default. At 15k RPS per PoP, we found that counting requests instead of sockets cut our p99 origin latency from 340ms to 92ms because we stopped penalizing servers holding idle keep-alives. You’ll also need to pick your polling interval carefully. If your load balancer only samples backend concurrency every 1-2 seconds, you’re making decisions on stale data during a traffic spike. We poll every 100ms and use an EWMA with a 500ms decay — fast enough to react, smooth enough to avoid flapping.

# Nginx - weighted least_conn with active health checks
upstream origin_cluster {
    least_conn;
    server origin-a.cdn.internal:443 weight=4 max_fails=2 fail_timeout=5s;
    server origin-b.cdn.internal:443 weight=2 max_fails=2 fail_timeout=5s;
    server origin-c.cdn.internal:443 weight=1 max_fails=2 fail_timeout=5s;
    keepalive 1024;
    keepalive_requests 10000;
}

# Envoy - least_request (P2C + least-connections) with slow-start
clusters:
- name: origin_cluster
  lb_policy: LEAST_REQUEST
  least_request_config:
    choice_count: 2  # Power of Two Choices - huge CPU saver vs scanning all hosts
  slow_start_config:
    slow_start_window: 30s
    aggression: 1.0
  health_checks:
  - timeout: 2s
    interval: 1s
    unhealthy_threshold: 3
    healthy_threshold: 2
    http_health_check: { path: "/healthz" }

Advanced Patterns and Troubleshooting: Herds, Draining, and Stale State

Once you’ve got weighted least-connections live, you’ll hit three classic failure modes. The first is the thundering herd on recovery. Say origin-b just came back from a 30-second GC pause. Its connection count is zero, so every edge node simultaneously decides it’s the “least loaded” and hammers it with 5,000 new requests in 200ms. It immediately OOMs again. That’s why you never deploy least-connections without slow-start. That 30-second slow_start_window in the Envoy config above linearly ramps the weight — we’ll send it 10% of traffic at 5s, 50% at 15s, and full weight only after 30s. It’s the difference between a clean recovery and a cascading failure.

The second gotcha is connection-count drift. If you’re not handling connection draining properly, your balancer thinks a server is full when it’s actually draining 800 lingering WebSockets that are no longer serving traffic. Pair your load balancing with proper administrative draining and passive feedback. We combine active polling with Active vs Passive Health Checks for CDN: Probing vs Observation-Based Monitoring — if a server starts returning 500s or its response latency spikes above 450ms for three consecutive requests, we temporarily inflate its connection count by 50% even if the active check still says “healthy.” For deeper probe design, you’ll want to look at Health Check Design Patterns for CDN: Shallow, Deep, and Synthetic Checks to avoid marking a server healthy when its cache disk is actually full.

Finally, know when *not* to use least-connections. If your CDN needs strict cache affinity — where object A must always hit the same origin to keep your hit ratio above 92% — least-connections will destroy your cache efficiency by scattering requests. In that case, you’re trading a 10ms latency improvement for a 40% origin offload loss. For those workloads, we use a Consistent Hash Ring Architecture for CDN: Minimizing Redistribution on Server Changes for cacheable GETs and reserve least-connections strictly for dynamic, non-cacheable API and POST traffic. Don’t try to make one algorithm do both jobs. Split your routing: hash for cache, least-connections for compute.

Frequently Asked Questions