Load Balancing: Algorithms, Architecture, and CDN Integration

Intermediate

No single server, however powerful, can handle the traffic demands of a modern web application. When millions of users simultaneously request pages, submit forms, and stream media, the workload must be distributed across a fleet of servers working in concert. Load balancing is the discipline of distributing this traffic intelligently — ensuring high availability, maximizing throughput, and minimizing response time. It is one of the foundational building blocks of scalable internet infrastructure.

This article explores the two primary architectural approaches to load balancing (Layer 4 and Layer 7), the algorithms that determine how traffic is distributed, health checking mechanisms that maintain reliability, and how CDN providers implement Global Server Load Balancing to route users to optimal edge servers worldwide.

How Did Load Balancing Evolve from Hardware to Software?

The history of load balancing traces the broader arc of internet infrastructure from specialized hardware to programmable software. In the early 1990s, as websites began receiving traffic volumes that exceeded the capacity of single servers, engineers developed ad-hoc solutions: DNS round robin (distributing traffic by rotating IP addresses in DNS responses) and manual server rotation. These approaches were crude — DNS round robin had no awareness of server health or load, and DNS caching made changes slow to propagate.

The first dedicated hardware load balancers appeared in the mid-1990s. F5 Networks released the BIG-IP product in 1997, and Cisco introduced the LocalDirector in 1996 and the Content Services Switch (CSS) in 2000. These were purpose-built appliances with custom ASICs designed to inspect and forward packets at wire speed. They supported advanced features like SSL offloading, session persistence, and health monitoring, but they were expensive (tens to hundreds of thousands of dollars) and vendor-locked.

The software revolution began with HAProxy, created by Willy Tarreau in 2000 and released as open source in 2001. HAProxy proved that commodity hardware running optimized software could match or exceed the performance of dedicated appliances at a fraction of the cost. nginx, released by Igor Sysoev in 2004, combined HTTP serving with reverse proxy and load balancing capabilities, further commoditizing the space. Today, software load balancers dominate, with cloud-native solutions like AWS Application Load Balancer, Google Cloud Load Balancing, and envoy proxy (created at Lyft in 2016, now a CNCF graduated project) handling traffic at unprecedented scale.

What Is the Difference Between L4 and L7 Load Balancing?

Load balancers are categorized by the OSI layer at which they make routing decisions. This distinction has profound implications for performance, capability, and appropriate use cases.

Layer 4 load balancing

L4 load balancers operate at the transport layer (TCP/UDP). They see source and destination IP addresses and port numbers but do not inspect the content of packets. Routing decisions are made based solely on this limited information — the load balancer forwards entire TCP connections or UDP flows to backend servers without understanding the application protocol being carried.

The primary advantage of L4 load balancing is raw performance. Because there is no protocol parsing, no content inspection, and no SSL termination, L4 load balancers can process millions of connections per second with minimal latency overhead. Linux's IPVS (IP Virtual Server), implemented in kernel space, can handle over 100 million concurrent connections on a single machine. Maglev, Google's custom L4 load balancer described in their 2016 paper, distributes traffic across their global infrastructure at 10 Gbps per machine.

L4 load balancing is ideal for protocols beyond HTTP: database connections (MySQL, PostgreSQL), message queues (AMQP, Kafka), gaming servers, and any TCP/UDP-based service where content-aware routing is unnecessary.

Layer 7 load balancing

L7 load balancers operate at the application layer, fully understanding HTTP (and often other application protocols like gRPC). They parse request headers, inspect URLs, read cookies, and can even examine request bodies. This deep visibility enables sophisticated routing capabilities that L4 load balancers cannot provide.

Content-based routing directs requests to different backend pools based on the URL path. For example, /api/* requests route to API servers while /static/* requests route to asset servers. Host-based routing uses the HTTP Host header to serve multiple domains from a single load balancer. Cookie-based session persistence ensures a user's subsequent requests reach the same backend server, maintaining session state. Header manipulation allows the load balancer to add, modify, or remove HTTP headers — injecting client IP addresses (X-Forwarded-For), adding request IDs for tracing, or setting security headers.

The trade-off is performance overhead. Parsing HTTP, terminating SSL/TLS, and maintaining application-layer state consumes more CPU and memory. An L7 load balancer typically handles tens of thousands to hundreds of thousands of requests per second — orders of magnitude fewer than a pure L4 solution. However, for web applications, the routing intelligence provided by L7 load balancing is almost always worth the overhead.

What Load Balancing Algorithms Exist and When Should Each Be Used?

The choice of load balancing algorithm determines how incoming requests are mapped to backend servers. Each algorithm makes different trade-offs between simplicity, fairness, and performance.

Round Robin is the simplest algorithm: requests are distributed to servers sequentially in a circular pattern. Server 1 gets request 1, server 2 gets request 2, server N gets request N, then back to server 1. Round robin assumes all servers have equal capacity and all requests impose equal load — assumptions that are rarely true in practice. It works well for stateless services with homogeneous server pools.

Weighted Round Robin addresses the heterogeneous server problem by assigning each server a weight proportional to its capacity. A server with weight 3 receives three times as many requests as a server with weight 1. This is useful when a pool contains servers with different hardware specifications — a 64-core machine should receive more traffic than an 8-core machine.

Least Connections routes each new request to the server with the fewest active connections. This algorithm naturally adapts to varying request complexity — a server processing a slow database query accumulates connections and receives fewer new requests. It is particularly effective for long-lived connections (WebSockets, streaming) where connection counts are a meaningful proxy for load. Weighted Least Connections combines connection counting with server weights.

IP Hash computes a hash of the client's IP address and maps it to a server. This provides natural session affinity — the same client always reaches the same server — without cookies or other application-layer mechanisms. The downside is that large NAT deployments (corporate offices, mobile carriers) can create hot spots where thousands of users behind a single public IP all hit the same backend.

Consistent Hashing is the most sophisticated algorithm and is widely used in CDN caching systems. Servers and request keys (URLs, for example) are both mapped onto a virtual hash ring. Each request is handled by the next server clockwise from its position on the ring. The critical advantage is stability: when a server is added or removed, only approximately 1/n of requests are remapped (where n is the number of servers), rather than reshuffling the entire mapping. This is essential for cache efficiency — remapping a request to a different server means losing the cached response on the original server.

The Ketama algorithm (originally developed at Last.fm) and Google's jump consistent hash are popular implementations. Modern refinements include bounded-load consistent hashing, which limits the maximum load any server can receive to prevent hot spots while maintaining the minimal-redistribution property.

How Do Health Checks Maintain Reliability?

A load balancer is only useful if it routes traffic to servers that are actually functioning. Health checking is the mechanism by which load balancers detect server failures and remove unhealthy servers from the rotation.

Active health checks are probes sent by the load balancer at regular intervals. An L4 health check might simply open a TCP connection to the server's port and verify it accepts connections. An L7 health check sends an actual HTTP request (typically to a dedicated /health endpoint) and validates the response status code, body content, and response time. Active checks can detect failures proactively but add monitoring traffic and may not capture all failure modes.

Passive health checks (also called real-user monitoring) observe the results of actual traffic. If a server returns a configurable number of errors (5xx responses, connection timeouts) within a time window, it is marked unhealthy. Passive checks detect failure modes that active checks might miss — a server might respond to health check probes while failing under real load — but they can only detect failures after real users have been affected.

Best practice is to combine both approaches. Active checks provide rapid detection of complete server failures (crash, network partition), while passive checks catch degraded performance and partial failures. The health check configuration specifies thresholds: how many consecutive failures trigger removal (typically 2-3), how many consecutive successes trigger re-insertion (typically 3-5), and the check interval (typically 5-30 seconds).

How Do CDNs Implement Global Server Load Balancing?

CDN providers face a unique load balancing challenge: routing users to the optimal edge server across hundreds of globally distributed locations. This problem is solved by Global Server Load Balancing (GSLB), which operates at a higher level than traditional load balancing.

GSLB typically uses DNS-based routing as its primary mechanism. When a user resolves the CDN's domain name, the GSLB controller (which serves as the authoritative DNS server) examines the user's IP address (or the EDNS Client Subnet, per RFC 7871), consults its real-time health and load data, and returns the IP address of the optimal PoP. The DNS response TTL is kept short (30-60 seconds) to allow rapid rerouting when conditions change.

Many CDNs complement DNS-based GSLB with anycast routing, where the same IP address is advertised from multiple PoPs via BGP. The internet's routing infrastructure naturally directs packets to the nearest announcement point. Anycast provides instant failover (BGP withdrawal is faster than DNS TTL expiry) and resilience against DDoS attacks (traffic is naturally absorbed across many locations).

What Is the Two-Tier Load Balancing Architecture?

Within each CDN PoP, a second tier of load balancing distributes traffic across individual edge servers. This local load balancing operates at Layer 7 and uses algorithms tuned for cache efficiency.

The most common approach combines consistent hashing on the request URL (to maximize cache hit rates by routing the same URL to the same server) with load-aware rebalancing (to prevent hot content from overwhelming a single server). When a particular piece of content becomes extremely popular, the local load balancer recognizes the hot key and begins spreading requests across multiple servers, accepting the cache duplication as a trade-off for preventing overload.

This two-tier architecture — GSLB for global routing and local load balancing within PoPs — gives CDNs the flexibility to optimize independently at each level. GSLB optimizes for proximity and availability; local load balancing optimizes for cache efficiency and server utilization.

How Do Modern Load Balancers Handle Session Persistence?

Many web applications require that consecutive requests from the same user reach the same backend server — a requirement called session persistence or session affinity. Shopping carts, multi-step forms, and applications that store session state in server memory all require this capability.

L7 load balancers implement session persistence through several mechanisms. Cookie-based persistence inserts a cookie identifying the target backend server. On subsequent requests, the load balancer reads this cookie and routes accordingly. This is the most reliable approach because it survives IP address changes (common on mobile networks). Source IP persistence maps client IP addresses to backends but fails when multiple users share an IP (NAT, proxies). Application-controlled persistence uses session IDs embedded in the URL or a custom header.

Modern architectures increasingly avoid session persistence by storing session state externally (in Redis, Memcached, or a database). This makes any backend server capable of handling any request, simplifying load balancing and improving resilience — if a server fails, no sessions are lost.

What Metrics Should You Monitor for Load Balancer Health?

Effective load balancer monitoring tracks metrics at multiple levels. Traffic metrics include requests per second, active connections, and bandwidth utilization — these reveal capacity trends and anomalies. Performance metrics include response time percentiles (p50, p95, p99) and backend response time versus total response time — the difference reveals load balancer overhead. Reliability metrics include error rates (4xx and 5xx by backend), health check failure frequency, and failover event count.

The most actionable metric is often backend response time distribution. A widening gap between p50 and p99 latency indicates that some requests are experiencing significantly worse performance — often a sign of uneven load distribution, a degraded server, or resource contention that the load balancing algorithm should address.