Edge Servers: How CDN Nodes Handle Millions of Requests

A CDN edge server is a specialized machine deployed at a Point of Presence whose sole purpose is to cache content and serve it to nearby users at maximum speed. These servers are the workhorses of every CDN, handling the overwhelming majority of end-user requests without involving the origin server. Understanding their hardware, software, and request-handling architecture reveals why modern CDNs can deliver content to billions of users with sub-20-millisecond latency.

What Hardware Powers a CDN Edge Server?

Edge servers are not generic cloud instances. They are purpose-built machines designed for one workload: reading cached data from storage and pushing it onto the network as fast as the interface allows. The hardware choices reflect this singular focus.

CPU

Modern edge servers typically run dual AMD EPYC 9004-series or Intel Xeon Scalable processors, providing 64-128 cores total. The CPU handles TLS encryption/decryption, HTTP parsing, compression, and connection management. Unlike compute-intensive workloads, edge serving is I/O-bound, so core count and memory bandwidth matter more than clock speed. Hardware AES-NI instructions accelerate TLS operations so that encryption runs at line rate.

Memory

RAM is the fastest cache tier. Edge servers typically carry 256-512 GB of DDR5 memory, with a portion reserved for the operating system and application buffers and the remainder allocated to the hot cache. Content that is accessed more than a few times per second stays in RAM, eliminating disk I/O entirely. This hot tier typically serves 60-80% of all requests by volume despite holding only 5-10% of the total cached content by size.

Storage

The warm cache tier runs on NVMe SSDs. A single edge server might hold 8-24 drives totaling 30-200 TB of flash storage. NVMe drives deliver 500,000-1,000,000 random read IOPS and 7+ GB/s sequential read throughput, enabling thousands of concurrent file reads without bottlenecking. Some providers add a cold tier using larger, less expensive SATA SSDs for long-tail content that is accessed only a few times per day.

Network

Network interface cards are the critical throughput bottleneck. Edge servers typically use dual 25 Gbps or 100 Gbps NICs, often with SmartNIC offloading for TLS, TCP segmentation, and checksum computation. High-end deployments at major peering locations may use 400 Gbps interfaces.

What Software Stack Runs on Edge Servers?

The software stack is equally purpose-built, combining an operating system, reverse proxy, caching engine, and supporting services into a lean, high-performance package.

Operating System

Most CDN edge servers run a hardened, minimal Linux distribution. Kernel tuning is extensive: increased file descriptor limits (millions per process), optimized TCP buffer sizes, enabled TCP Fast Open, tuned interrupt coalescing on NICs, and CPU affinity pinning for network threads. Some providers compile custom kernels with only the modules required for edge operation.

Reverse Proxy and Cache Engine

The core application is a reverse proxy that handles HTTP/HTTPS request processing and caching. The most common choices are:

• NGINX — Event-driven architecture, handles 100K+ concurrent connections per worker process. Used by many mid-size CDN operators.
• Apache Traffic Server (ATS) — Open-source, battle-tested at scale by Apple, Yahoo, and Comcast. Excellent hierarchical caching support.
• Varnish Cache — In-memory caching with the VCL (Varnish Configuration Language) for flexible cache logic. Often used in front of other layers.
• Custom implementations — Cloudflare uses a custom Rust-based proxy (Pingora), Fastly runs their Compute platform on a custom Rust stack, and Akamai operates proprietary edge software.

TLS Library

TLS is handled by libraries like OpenSSL, BoringSSL (Google fork), or rustls. The library manages certificate storage, session resumption, OCSP stapling, and cipher negotiation. Proper TLS configuration at the edge is critical because every HTTPS connection begins with a handshake whose latency depends on the distance between client and server.

Supporting Services

Edge servers also run health check agents (reporting status to the load balancer), log shipping daemons (sending access logs to central analytics), configuration sync agents (pulling updated rules from the control plane), and metrics collectors (CPU, memory, disk, network utilization).

How Does an Edge Server Handle a Request?

When a user requests a resource, the edge server executes a precise sequence of operations that typically completes in under 5 milliseconds for cached content:

1. TCP connection — The client establishes a TCP connection (or reuses an existing one via HTTP/2 or HTTP/3 multiplexing). The edge server SYN-ACK completes in under 1 ms for local users.
2. TLS handshake — If HTTPS, the server performs the TLS handshake using its certificate. Session resumption (TLS 1.3 0-RTT) skips this step for returning visitors, saving a full round trip.
3. HTTP request parsing — The server parses the HTTP request, extracting the method, URL, headers, and any query parameters.
4. Cache key generation — A cache key is computed from the URL and relevant headers (Vary, Accept-Encoding). This key determines which cached object to look up.
5. Cache lookup — The server checks RAM first (sub-microsecond), then NVMe SSD (50-200 microseconds). A cache hit proceeds directly to response; a cache miss triggers an origin fetch.
6. Response construction — The server adds response headers (Cache-Control, Age, X-Cache), applies compression if needed, and streams the response body.
7. Logging — The request is logged asynchronously with timing data, cache status, client IP, and response code for later analytics.

How Does Connection Pooling Improve Performance?

Connection pooling is an essential optimization between edge servers and upstream tiers (or the origin). Without pooling, every cache miss requires a new TCP+TLS connection to the origin, adding 100-500 ms of latency. With persistent connection pools, edge servers maintain a set of pre-established connections to origin servers and reuse them across multiple requests.

A well-tuned edge server maintains pools of 50-200 persistent connections to each origin, with idle timeout values of 60-300 seconds. When a cache miss occurs, the request is routed over an existing connection, eliminating the handshake overhead entirely. This is especially impactful for edge computing workloads where dynamic content requires frequent origin communication.

How Is Cache Storage Organized on an Edge Server?

Cache storage on an edge server is organized into tiers based on access speed and cost, following a principle similar to CPU cache hierarchy:

Content promotion between tiers happens automatically. An object requested for the first time is fetched from upstream and stored in L2. If it receives repeated requests within a short window, it is promoted to L1 (RAM). Objects that have not been accessed within their TTL are evicted, starting from L1 and working down. Eviction algorithms vary: LRU (Least Recently Used) is common, but some providers use LFU (Least Frequently Used) or ARC (Adaptive Replacement Cache) for better hit ratios on mixed workloads.

How Many Requests Can a Single Server Handle?

Capacity depends heavily on the workload profile. Here are realistic benchmarks for a modern edge server with the specifications described above:

• Small objects (1-10 KB, API responses, CSS) — 200,000-500,000 requests per second, bottlenecked by CPU and connection handling
• Medium objects (100 KB - 1 MB, images) — 50,000-150,000 requests per second, balanced between CPU and disk I/O
• Large objects (10-100 MB, video segments) — 1,000-10,000 requests per second, bottlenecked by network bandwidth (100 Gbps = ~12.5 GB/s)
• Concurrent connections — 500,000-2,000,000 simultaneously maintained TCP connections

These numbers explain why a PoP with just 10-20 servers can serve an entire metropolitan area. Ten servers handling 200,000 requests per second each provides 2 million requests per second of aggregate capacity, sufficient for all but the largest traffic events.

What Role Does TLS Termination Play at the Edge?

TLS termination at the edge is one of the most significant performance optimizations a CDN provides. A TLS 1.3 handshake requires one round trip between client and server. If the client in Mumbai connects to an origin in Frankfurt (160 ms RTT), the handshake alone takes 160 ms. Terminating TLS at a Mumbai edge server (5 ms RTT) reduces the handshake to 5 ms, a 97% improvement before any content is even delivered.

Modern edge servers handle TLS at line rate using hardware AES-NI instructions and dedicated CPU cores pinned to TLS operations. A single server can perform 100,000+ TLS handshakes per second while simultaneously serving cached content. Session tickets and TLS 1.3 0-RTT resumption further reduce overhead for returning visitors, eliminating the handshake entirely for repeat connections.

The connection between edge and origin can use a separate, persistent TLS connection (mTLS in enterprise deployments), but this connection is established once and reused across thousands of requests through connection pooling, amortizing the handshake cost to near zero.

Edge servers represent the engineering core of any CDN. Their hardware is selected for I/O throughput, their software is tuned for concurrent connection handling, and their cache architecture is tiered for optimal hit ratios. Understanding these internals helps you make better decisions about CDN architecture, cache configuration, and performance optimization for your own applications.