TCP Health Checks for CDN: Layer 4 Monitoring When HTTP Isn't Enough

About three years ago I got paged at 1:47 AM because an entire PoP in Frankfurt was returning 502s, yet our dashboard showed every origin as "healthy." The HTTP health checks were all passing — 200 OK on /health — but the application behind it had deadlocked and couldn't serve real traffic. The TCP stack was still answering, the HTTP endpoint was still returning a cached 200, and our Layer 7 checks were happily marking a dead origin as alive. That's when we switched that origin pool to aggressive HTTP Working Group Specifications compliant TCP health checks, and we caught the failure 40 seconds earlier on the next incident.

Why HTTP Health Checks Alone Will Lie to You

HTTP health checks are great when you need to know if your application logic is working, but they’re surprisingly easy to fool. I’ve spent a decade running CDN origins where the health endpoint is handled by a completely different code path than customer traffic. You might have nginx returning 200 for /healthz while your upstream app servers are hung, your PHP-FPM pool is exhausted, or your TLS certificate rotation has broken real handshakes. Layer 7 checks don’t just test connectivity — they test an entire software stack, and that stack can lie. A CDN L4 monitoring approach strips that away and asks the only question that matters at the transport layer: can I actually open a TCP connection to this box?

There’s also the overhead problem. An HTTP check requires a full three-way handshake, plus a request, plus waiting for headers and body. On a CDN with thousands of origins per edge, that adds up fast. I worked with a video streaming customer last year who had 800 origins behind their GSLB Implementation Patterns: Global Server Load Balancing for Multi-Region CDN setup. Their HTTP checks every 5 seconds were generating nearly 14k HTTP requests per minute just for monitoring. Switching half of those pools to a simple TCP connection check cut their monitoring bandwidth by over 60% and let them halve the interval without adding load. That’s the trade-off you’re buying with Layer 4 — it’s faster, lighter, and brutally honest about network reachability.

That doesn’t mean you should replace every HTTP check. I still use HTTP checks for origin pools where application correctness matters, like API clusters that depend on database connectivity. But for raw origin pools, edge caches, or any setup where you’re using Weighted Round-Robin for CDN: Proportional Traffic Distribution Across Heterogeneous Servers, a TCP health check belongs as your first line of defense. It catches the network partitions, firewall misconfigurations, kernel panics, and SYN flood blackholes that HTTP checks either miss or catch too slowly. For insights on how modern browsers handle these connection failures, the web.dev Articles provide excellent context on connection resilience.

# HAProxy - Compare L7 vs L4 health checks
# L7 HTTP check - thorough but heavier and can be fooled
backend origins_http
  option httpchk GET /healthz
  http-check expect status 200
  server origin1 192.0.2.10:443 check inter 5s fall 3 rise 2 ssl verify required
  server origin2 192.0.2.11:443 check inter 5s fall 3 rise 2 ssl verify required

# L4 TCP check - lightweight connection check
backend origins_tcp
  option tcp-check
  tcp-check connect port 443
  server origin1 192.0.2.10:443 check inter 2s fall 2 rise 2
  server origin2 192.0.2.11:443 check inter 2s fall 2 rise 2
  # Fails fast if SYN-ACK never comes back - no HTTP parsing involved

How a TCP Connection Check Works Under the Hood

A TCP health check is deceptively simple, which is exactly why it’s reliable. The CDN edge or load balancer initiates a standard TCP three-way handshake: it sends a SYN, waits for a SYN-ACK, then sends an ACK. If that completes, the check passes and it immediately sends a RST or FIN to tear the connection down — no data is ever sent. That’s it. There’s no HTTP request, no TLS handshake (unless you explicitly configure a TLS-aware TCP check), and no application logic in the way. I tell junior engineers to think of it as knocking on the door to see if someone’s home, rather than asking them to recite their name and address.

What you’re really testing here is the entire Layer 4 path. This catches failures that sit below the application: a security group change that blocks port 443, an origin server whose conntrack table is full, a kernel that’s stopped accepting new sockets due to file descriptor exhaustion, or a BGP flap that blackholes traffic to a specific rack. I once debugged an incident where a customer’s origins were failing only for users in Southeast Asia. HTTP checks from our US-based prober looked fine, but TCP checks from the Singapore PoP failed 40% of the time due to an MTU issue on a transit provider that only manifested during the SYN-ACK. A connection check from the actual edge caught it; a centralized HTTP check never would have.

You can configure how deep the check goes. The most basic form is a SYN-only check that passes the moment a SYN-ACK is received. A more thorough version completes the handshake and optionally sends a short probe payload to ensure the socket stays open. For CDNs using Least-Connections Load Balancing for CDN: Dynamic Distribution Based on Server Load, pairing TCP checks with least-connections logic is particularly effective — you don’t want to send new connections to an origin that can’t even complete a handshake, especially when you’re trying to balance based on active connection count. The trick is to not overthink it: keep the probe payload empty unless you’re specifically testing for a banner or TLS handshake.

// Node.js - Minimal TCP health check prober (what your CDN does internally)
const net = require('net');

function tcpHealthCheck(host, port, timeoutMs = 3000) {
  return new Promise((resolve) => {
    const socket = new net.Socket();
    let status = 'failed';

    const timer = setTimeout(() => {
      socket.destroy();
      resolve({ host, port, status: 'timeout', latencyMs: timeoutMs });
    }, timeoutMs);

    const start = Date.now();
    socket.connect(port, host, () => {
      const latencyMs = Date.now() - start;
      status = 'open';
      clearTimeout(timer);
      socket.end(); // Send FIN - clean close after successful ACK
      resolve({ host, port, status, latencyMs });
    });

    socket.on('error', (err) => {
      clearTimeout(timer);
      // ECONNREFUSED, EHOSTUNREACH, timeout all mean L4 failure
      resolve({ host, port, status: 'failed', error: err.code });
    });
  });
}

// Usage: probe every 2s from each edge PoP
setInterval(async () => {
  const result = await tcpHealthCheck('192.0.2.10', 443);
  console.log(result); // { host: '192.0.2.10', port: 443, status: 'open', latencyMs: 12 }
}, 2000);

Tuning Check Intervals, Timeouts, and Failure Thresholds

Getting the timing right is where most teams mess this up. I’ve seen configs with 30-second intervals and a 10-second timeout that took nearly two minutes to mark a dead origin as down — during which the CDN kept sending customer traffic into a black hole. On the other end, I’ve seen 500ms intervals with a 1-second timeout that flapped constantly and caused more problems than they solved. For edge-facing CDN L4 monitoring, you want checks frequent enough to fail fast, but not so aggressive that you create a SYN flood against your own origins.

My baseline for production CDN origin pools is a 2-second interval, a 3-second timeout, with 2 consecutive failures to mark down and 2 successes to mark up. That gives you a 4-6 second detection window, which is a sweet spot I’ve landed on after a lot of trial and error. If your origin is behind an anycast IP or you’re running health checks from dozens of PoPs, you’ll want to stagger the checks so they don’t all hit at once. Add a small jitter — say 200-400ms random delay — to avoid thundering herd effects. I learned this the hard way when 45 PoPs all probed the same origin at the exact same second and tripped its DDoS protection.

Here’s how I usually map check tuning to different origin types. Don’t treat these as gospel, but they’re a solid starting point before you tune based on your own latency percentiles. Track your p95 and p99 SYN-ACK times for a week before you lock these in — your timeout should be at least 3x your p99, or you’ll be chasing ghosts.

Putting TCP Checks Into Production: Configuration That Won't Flap

Alright, theory's great — but let's get this running without taking down your origin pool at 2 a.m. The biggest mistake we see teams make is copy-pasting the default health check values from their load balancer docs. Defaults like a 10-second interval with a 30-second timeout are way too slow for a modern CDN edge. If an origin dies, you don't want to keep hammering it with user traffic for half a minute while your PoPs figure it out.

For most high-traffic origin pools, we recommend starting much more aggressively and then backing off if you see noise. Try a 3-5 second interval, a 2-second connect timeout, a rise threshold of 2 successes to mark healthy, and a fall threshold of 3 consecutive failures to mark unhealthy. That gives you a 6-9 second detection window for a hard failure, which is fast enough to fail over before users notice, but not so sensitive that a single packet drop flaps the whole pool. If you're running IP Hash for CDN Session Persistence: Consistent User Routing Without Cookies, this matters even more — a flapping check will constantly remap users to different origins and break persistence.

And keep the check lightweight. A TCP check should be a bare SYN -> SYN-ACK -> RST, not a full TLS handshake. Don't hold the connection open. On a fleet of 150 PoPs checking 20 origins every 3 seconds, you're looking at 1,000 check SYNs per second per origin. That's nothing for a tuned kernel, but it'll crush you if each check tries to negotiate TLS 1.3. Here's what a sane config looks like in HAProxy or NGINX Plus:

# HAProxy - Layer 4 TCP check with sane thresholds
backend origin_pool
  option tcp-check
  tcp-check connect port 443
  default-server inter 3s rise 2 fall 3 connect-timeout 2s
  server origin1 203.0.113.10:443 check
  server origin2 203.0.113.11:443 check
  server origin3 203.0.113.12:443 check backup

# NGINX Plus equivalent
upstream origin_pool {
  zone origin_pool 64k;
  server 203.0.113.10:443 max_fails=3 fail_timeout=9s;
  server 203.0.113.11:443 max_fails=3 fail_timeout=9s;
}

# Plus health_check directive in location (NGINX Plus)
# health_check interval=3s fails=3 passes=2 port=443;

Advanced Patterns and Troubleshooting When TCP Lies

Even TCP checks can lie if you don't understand what you're actually measuring. The most common gotcha? Your firewall is healthier than your origin. We've debugged cases where a SYN flood mitigation rule was happily ACKing every SYN on behalf of an origin that had completely locked up — kernel panic, but the security group still responded. If you can, terminate the TCP check on the actual origin host, not an intermediate NLB or firewall. And watch your conntrack tables. On busy origins handling 40k+ RPS, we've seen health checks get dropped because nf_conntrack_max was set to the default 65k and the table was full. The CDN thought the origin was down, but it was just out of conntrack slots.

That doesn't mean you should ditch HTTP checks entirely. The best pattern we've deployed at scale is a layered approach — use TCP as your fast, cheap gatekeeper and HTTP as your slower, deeper verifier. If TCP fails, pull the origin immediately. If TCP passes but your shallow HTTP check on /healthz starts returning 500s or 5-second latencies, then pull it for application-level reasons. This two-tier model is covered in detail in our guide to Health Check Design Patterns for CDN: Shallow, Deep, and Synthetic Checks. It saves you from the classic case where TCP says "port's open!" but the app server behind it is returning 503s to every real user.

One more thing for large, stateful pools: combine TCP check aggressiveness with your hashing strategy. If you're using a Consistent Hash Ring Architecture for CDN: Minimizing Redistribution on Server Changes, a single origin flapping can cause massive cache redistribution if you mark it down too quickly. In that setup, we often bump fall to 4 or 5 and add a 30-second slow-start when a node rejoins. That way you don't yank 1/Nth of your cached objects off the ring because of two dropped SYNs during a brief network blip. Graceful draining is your friend — mark failed nodes as draining, not dead, and let existing connections finish.

Frequently Asked Questions