Origin Connection Pooling Optimization: Reducing TCP Overhead at CDN Scale

Back in 2019, right before Black Friday traffic peaked, we watched our origin QPS flatline while CDN edge CPU sat at 40%. It wasn't the application — it was TCP. We were opening and closing nearly 12,000 new connections per second to a pool of six origins, and the handshake overhead, TIME_WAIT sockets, and TLS negotiation were choking the whole path. That incident forced us to rethink how we pooled origins, and it cut our origin latency in half overnight.

Why Per-Request Connections Fall Apart at Scale

When you're running a few hundred requests per second, opening a fresh TCP connection to origin for each cache miss doesn't feel expensive. The kernel handles it, your latency graphs look fine, and you move on. But once you hit CDN scale — thousands of edge nodes, each fanning out to a small set of origins — that model breaks down fast. Every new connection pays the three-way handshake tax, plus TLS if you're doing what you should be doing, plus slow-start. I've measured 30-45ms of pure connection setup on cross-region origins before a single byte of HTTP even goes out.

What makes it worse is that most of that work is completely wasted. Origin traffic is bursty and highly reusable. If you looked at our logs during that 2019 event, 88% of origin requests were to the same 20 URLs, hitting the same keep-alive-capable servers. We weren't reusing anything. Linux was spending more time in connection teardown than in data transfer, and our origins were holding thousands of sockets in TIME_WAIT. You can see that pathology clearly in ss -s and netstat -an | grep TIME_WAIT output when you're not pooling — it's not subtle. Pairing connection reuse with solid Origin Health Monitoring Patterns: Active Probes, Passive Checks, and Hybrid Approaches helps you avoid sending recycled connections to a sick host, which is a failure mode I've seen teams miss when they first enable keep-alive.

The HTTP specs themselves assume you'll reuse connections. The HTTP Working Group Specifications (https://httpwg.org/specs/) describe persistent connections as the default behavior in HTTP/1.1 precisely because the cost of setup is so high relative to the request itself. And guidance on web.dev Articles (https://web.dev/articles/) around connection efficiency consistently shows that eliminating handshakes is one of the highest-leverage things you can do for time-to-first-byte. At the edge, we treat a connection as infrastructure, not as a per-request disposable.

# Nginx upstream keepalive - the baseline for origin pooling
# Without this, nginx opens a new TCP connection per proxied request

upstream origin_backend {
    server origin1.example.com:443 max_fails=3 fail_timeout=30s;
    server origin2.example.com:443 max_fails=3 fail_timeout=30s;
    keepalive 64; # idle keepalive connections per worker
    keepalive_timeout 60s;
    keepalive_requests 10000; # requests per connection before close
}

server {
    location / {
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_pass https://origin_backend;
        # Critical: don't close on each request
        proxy_socket_keepalive on;
    }
}

Tuning Pool Size, Timeouts, and Sharding Without Starving Origins

There's a common mistake I see teams make after they enable keepalive: they set keepalive 1024 and assume bigger is better. It isn't. An oversized pool creates idle connections that hold memory on both edge and origin, trigger firewall idle timeouts, and then fail on the next reuse with a stale RST. We learned this the hard way when a well-meaning increase from 32 to 256 idle conns per worker actually increased our 5xx rate by 0.3% — the NAT gateway between edge and origin was silently dropping idle TCP after 120 seconds, and we were trying to reuse dead sockets.

The right size depends on your worker count, your origin concurrency, and your miss rate. A practical starting point we've used across multiple CDN pops is: pool_per_worker = (peak_origin_rps_per_node / workers) * target_keepalive_ratio. For example, an edge node doing 800 origin RPS with 8 workers might aim for 16-32 idle connections per worker, not 100. You want enough to absorb bursts without handshakes, but not so many that most sit idle past your network's idle timeout. You also need to shard pools by origin IP and consider per-host pools if you route to multiple origin clusters — a single global pool can pin you to one origin and hide imbalance.

Timeout tuning matters just as much as size. Set your keepalive_timeout a few seconds below the lowest timeout in the path — typically your load balancer or origin server's keepAliveTimeout. If your origin keeps idle connections for 60s but your ELB kills them at 60s, set edge to 55s. And set a keepalive_requests ceiling high, but not infinite; recycling after 5k-10k requests lets you gracefully rebalance without mass connection churn. This interacts directly with your retry logic, so if you haven't already, review your Origin Retry Strategies for CDN Resilience: Backoff, Jitter, and Budget Limits to make sure a failed reuse doesn't turn into a retry storm on an already strained origin.

// Envoy cluster config with tuned connection pooling
// Focus on max_requests_per_connection and idle timeout alignment

clusters:
- name: origin_cluster
  connect_timeout: 0.25s
  max_requests_per_connection: 10000
  circuit_breakers:
    thresholds:
      - max_connections: 10000
        max_pending_requests: 1000
        max_requests: 8000
  upstream_connection_options:
    tcp_keepalive:
      keepalive_time: 45 # seconds, below LB/origin 60s timeout
      keepalive_interval: 10
      keepalive_probes: 3
  common_http_protocol_options:
    idle_timeout: 55s # must be < origin idle timeout

// Application-level pool health check (Node.js / undici example)
import { Pool } from 'undici';

const pool = new Pool('https://origin.example.com', {
  connections: 32, // per origin, per worker
  pipelining: 1, // 1 for origin safety, avoid head-of-line blocking
  keepAliveTimeout: 55000, // 55s - 5s below origin
  keepAliveMaxTimeout: 60000,
});

What We Actually Measure to Prove Pooling Works

Don't trust that pooling is working because you turned it on. Measure reuse. The two numbers I watch first are connection reuse ratio (reused_connections / total_origin_requests) and TCP handshake rate per second. On a healthy pooled node, reuse should be above 90% and handshake rate should be an order of magnitude lower than request rate. When we fixed that 2019 incident, reuse went from 12% to 96% and handshakes dropped from ~12k/sec to ~400/sec on the same traffic. Origin CPU dropped 18% just from losing the TLS handshake load.

The second layer is tail latency. Pooling doesn't just save CPU — it shaves variance. New connections are subject to SYN retransmits, TLS negotiation jitter, and slow-start. Reused connections skip all of that. In our staging tests, p95 origin fetch time fell from 180ms to 62ms after pooling was tuned, while p50 only moved from 45ms to 38ms. That tail improvement is huge if you're considering Origin Request Hedging: Sending Parallel Requests to Cut Tail Latency later — you may find you don't need hedging at all once pooling is clean, or you can hedge far less aggressively and save the duplicate origin load.

Here's what a typical before/after looked like for one of our mid-tier pops (8 edges, 4 origins, ~5k origin RPS total) when we moved from per-request connections to tuned keepalive pools with proper timeout alignment:

Putting It Into Practice: A Production-Ready Pool Configuration

Theory's great, but pools live or die in the config file. If you're running NGINX as your edge proxy (and most of us still are, somewhere in the stack), the defaults will absolutely burn you. The stock keepalive 32 sounds reasonable until you're pushing 40k RPS per PoP and your origins are drowning in TCP handshakes. We've found that most teams under-provision keepalive by a factor of 3-5x because they're sizing for average load, not burst.

A practical starting point we use: pool_size = (peak_RPS_per_edge * p95_origin_response_time) * 1.5. So if a single edge node peaks at 5,000 RPS to origin and your p95 origin response is 80ms, you need roughly 600 idle keepalive connections per origin, not 32. Shard that across your origin IPs and you’ll keep TIME_WAITs sane. Pair that with aggressive but sane timeouts — we run a 500ms connect timeout, 5s read timeout, and 60s keepalive idle timeout. That 60s window is the sweet spot; anything longer and you’re holding dead sockets that the origin’s LB has already silently dropped. For a deeper dive on getting those three values right, see our Origin Timeout Configuration Guide: Connect, Read, and Total Timeout Tuning.

Don't forget TLS. Reusing a TCP socket means nothing if you're re-doing the full TLS handshake every time. You want TLS session resumption and you absolutely need proper certificate validation on that reused connection, otherwise you're just building a really efficient MITM highway. We enforce proxy_ssl_verify on and proxy_ssl_session_reuse on on every pooled connection, even if it costs us an extra 2-3ms on the first hit. The security trade-off isn't worth it. We cover the full verification flow in our Origin SSL Verification at CDN Edges: Preventing Man-in-the-Middle Attacks guide.

# Production-tested NGINX origin pool - per upstream block
upstream origin_backend {
    server origin.example.com:443 max_fails=3 fail_timeout=30s;
    keepalive 512;          # Not 32 - sized for 5k RPS * 80ms p95
    keepalive_timeout 60s;  # Match origin's keepalive_timeout + 5s
    keepalive_requests 10000; # Recycle before origin resets
}

server {
    location / {
        proxy_pass https://origin_backend;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        
        # Critical for pooled TLS
        proxy_ssl_verify on;
        proxy_ssl_trusted_certificate /etc/ssl/certs/ca-bundle.crt;
        proxy_ssl_session_reuse on;
        proxy_ssl_server_name on;

        # Timeouts aligned with pool
        proxy_connect_timeout 500ms;
        proxy_read_timeout 5s;
        send_timeout 5s;
    }
}

Advanced Patterns and Troubleshooting When Pools Misbehave

Once you've got the basics stable, you'll hit the next tier of problems. The first is hot-sharding. With consistent hashing, one origin IP can end up owning 40% of your pool because of a skewed URL hash. We moved to least-connections sharding with bounded loads and saw origin CPU variance drop from ±35% to ±8% overnight. If you're using HTTP/2 to origin, it's even trickier — you don't want 500 streams multiplexed on one TCP connection while another sits idle. We cap it at 100 streams per H2 connection and actively create new TCP connections once we hit 75. It’s a bit more overhead, but it prevents that single-connection head-of-line blocking that tanks your p99.

When pooling goes wrong, it fails in confusing ways. The classic symptom we see is a spike in 502s with upstream prematurely closed connection in the logs, but only at around 02:00 UTC. That's almost always an idle timeout mismatch — your CDN's keepalive_timeout 60s is longer than the origin ELB's 55s idle timeout, so the LB RSTs a connection your edge thought was alive. The fix isn't to just bump the timeout; you've got to set your edge 5-10s *lower* than the origin side and add a background health check that prunes stale sockets every 10s. Also watch your netstat -an | grep TIME_WAIT | wc -l. If you're correctly pooling, that number should stay under 5k per edge node. If it's at 30k+, your pool isn't actually reusing anything.

There's also the noisy neighbor problem, but at the IP layer. When you pool aggressively from a small set of edge IPs, origins (and their WAFs) can mistake you for a single abusive client. We had a client whose origin rate-limited its own CDN because 800 pooled connections from 4 IPs looked like a botnet. That's where origin-facing IP hygiene becomes critical. You need enough egress diversity and clean reputation, or your perfectly tuned pool will just get 429'd. We've documented our approach to rotating and warming those addresses in Origin IP Reputation Management: Keeping Your CDN Source IPs Clean. Combine that with circuit breakers — we trip at 50% 5xx for 10 seconds and drain the pool — and you won't cascade a failing origin across your entire fleet.

Frequently Asked Questions