Health Check Interval Optimization: Balancing Detection Speed and Resource Usage

Last Black Friday, we pushed our health check interval down to one second to catch failures faster — and within twenty minutes our origin monitors were drowning in check traffic, not real user requests. Edge CPU spiked 18% and we started getting false positives from a healthy PoP that was just slow to answer under load. That night taught me you can't just crank up check frequency and hope for the best; you've got to balance how quickly you detect a failure against what it actually costs to keep checking.

Why a 1-Second Check Can Do More Harm Than Good

When I started working on CDN load balancing about twelve years ago, the default advice was simple: check as often as you can afford. Most teams set their health check interval to 1-2 seconds, figured more data meant safer traffic, and called it a day. I've run that setup myself on a 60-node video edge and it feels great until you scale. Every check is a real HTTP request — it needs a TCP handshake, TLS if you're checking HTTPS origins, a full request/response cycle, and logging. Multiply that by hundreds of edges checking dozens of origins and you're not just monitoring health, you're generating a small DDoS against your own infrastructure.

The other problem nobody warns you about is noisy failure detection. If your timeout is tight and your interval is aggressive, a single slow response or a brief packet loss event looks like an outage. We've all seen flapping — a node gets marked down, traffic shifts, it recovers, traffic shifts back, then it flaps again. That churn is worse for cache hit ratio and origin load than just leaving a slightly degraded node in rotation for another five seconds. In one of our European PoPs, we were marking a storage origin as unhealthy three or four times an hour, but it wasn't actually down. The health checker was just too impatient.

What helped was treating health checks like any other control plane signal: they should be reliable, cheap, and not trigger on every blip. We moved to requiring consecutive failures before marking unhealthy, added jitter so checks don't thundering herd, and stopped using the same interval for every origin. Your hot cache tier and your cold archive origin shouldn't be checked the same way. Here's a pattern we now use as a baseline for HTTP origins — it's boring, but it cut our false positives by over 70%:

# baseline health check for CDN origin pool
# tuned for intermediate scale: ~80 edges, 12 origins
health_check:
  interval: 5s          # time between checks per target
  timeout: 2s           # fail if no response in 2s
  healthy_threshold: 2  # 2 consecutive successes to mark UP
  unhealthy_threshold: 3 # 3 consecutive failures to mark DOWN
  jitter: 0.5s          # randomize to avoid synchronized checks
  http:
    path: /cdn-health
    expected_status: 200
    expected_body: "ok"
    headers:
      User-Agent: "cdn-health-checker/1.0"
      Cache-Control: "no-cache"

Tuning Detection Speed Without Burning Your Control Plane

Detection speed isn't just the interval. It's interval multiplied by how many failures you require, plus timeout and any debounce you add. If you check every 5 seconds but need three failures to mark a node down, your worst-case detection is around 15 seconds, not 5. That's intentional. You want that window to filter out transient noise while still being fast enough that users don't hit errors for long. For most of our customer-facing edges, we aim for a 10-20 second detection window for edge-to-origin failures. It's quick enough that retry logic and failover can hide the problem from users, but not so quick that we're reacting to every hiccup.

I learned this the hard way on a GSLB Implementation Patterns: Global Server Load Balancing for Multi-Region CDN rollout where we tied global DNS failover directly to health checks. We had a 3-second interval with a single-failure threshold, and a brief upstream congestion event in Ashburn made us pull an entire region out of DNS. TTLs meant it took six minutes to recover even though the PoP was healthy again in 15 seconds. Now we decouple local load balancing decisions from global ones. Local pools can react in 10-15 seconds, but GSLB needs two consecutive windows of failure or a longer interval, typically 10 seconds with a 3-failure threshold, so about 30 seconds to fail a region.

The trick is to make check frequency adaptive, not static. Under normal conditions you don't need to hammer healthy nodes. Under degraded conditions you want faster feedback. We run a simple adaptive loop: if a node has been healthy for 10 minutes, we back off slightly; if it starts failing once, we tighten the interval temporarily until it's confirmed up or down. It doesn't need to be fancy. Compared to static Weighted Round-Robin for CDN: Proportional Traffic Distribution Across Heterogeneous Servers setups where every backend gets the same poll rate regardless of weight or recent history, adaptive checks save a ton of overhead. And if you're using Least-Connections Load Balancing for CDN: Dynamic Distribution Based on Server Load, you’ll notice health check noise directly skews your connection counts if you’re not careful — flapping nodes look like zero-load targets and get flooded when they come back.

// adaptive health check interval - Go-style pseudocode
// adjusts check frequency based on recent health
func nextInterval(state NodeState) time.Duration {
    baseInterval := 5 * time.Second
    if state.ConsecutiveFailures == 1 {
        // first failure - check faster to confirm
        return 2 * time.Second
    }
    if state.ConsecutiveSuccesses > 20 {
        // stable node - save resources, check less often
        return 10 * time.Second
    }
    if state.FlappingCount > 3 {
        // flapping - dampen checks to avoid churn
        return 15 * time.Second
    }
    // add jitter: +/- 15%
    jitter := time.Duration(rand.Float64()*0.3-0.15) * baseInterval
    return baseInterval + jitter
}

What Different Intervals Actually Cost in Production

Numbers make this trade-off concrete. We benchmarked four common health check interval settings on the same fleet: 80 cache nodes checking 12 origins over HTTP, each check a GET to /cdn-health with a 2-second timeout. We measured mean time to detect a hard failure (we killed the origin process), checks per minute, and the extra CPU and bandwidth cost just for health checking. The HTTP Working Group Specifications guidance on connection reuse and the caching best practices in web.dev Articles helped us keep checks efficient — we reuse connections where possible and avoid cache-busting without Cache-Control: no-cache, but even with keep-alive, frequency matters.

What stood out wasn't just overhead, it was the false positive rate. At 1-second intervals we were marking nodes unhealthy on slow responses that would have succeeded at 2 seconds, especially during origin GC pauses or brief CPU spikes. At 30 seconds, we barely noticed any false positives, but users saw errors for nearly a minute when an origin actually died before failover kicked in. For our workload — mixed video and API traffic — the 5-second interval with 3x failure threshold hit the best compromise: fast enough to keep p95 error rates low, quiet enough that we weren't wasting capacity. Your sweet spot will depend on your retry policy and how much stale traffic you can tolerate, but don't guess — measure detection speed against resource usage on your own fleet first.

Practical Implementation: Tuning Intervals Without Breaking Failover

Look, the theory's clean but production's messy. We've seen teams set a perfect 5-second interval in staging, then watch their control plane melt when they roll it to 800 edge PoPs. The trick isn't picking a number — it's implementing it with enough guardrails that you don't trade fast detection for flapping and false positives. Our baseline that works for 95% of CDN edge pools is this: 5s interval, 2s timeout, 3 consecutive failures to mark unhealthy, and 2 consecutive successes to bring it back. That gives you a 15-second detection window with a 10-second recovery - fast enough that users don't notice, slow enough that a single GC pause doesn't drain a whole origin.

Don't hardcode the same interval for every dependency, though. Your shallow L7 check on the edge cache itself can run every 5s — it's cheap, just a TCP + HTTP 200 on /cdn-health/shallow. Your deep synthetic that hits origin, warms cache, and validates TLS should be on a completely different schedule: 30s or even 60s. If you're using Health Check Design Patterns for CDN: Shallow, Deep, and Synthetic Checks correctly, you'll split them into two probes. Otherwise you'll be running a 400ms deep check every 5 seconds across 2,000 nodes and wondering why your origin's logging 24k synthetic requests per minute that aren't even real traffic.

Here's what that split looks like in Envoy — and the same logic maps directly to Nginx Plus or HAProxy. Note the jitter and the separate thresholds; without jitter, you'll get a thundering herd every 5.0s on the dot:

health_checks:
  - timeout: 2s
    interval: 5s
    interval_jitter: 0.5s  # critical: +/- 10% randomization
    unhealthy_threshold: 3 # 15s to eject
    healthy_threshold: 2   # 10s to re-add
    http_health_check:
      path: "/cdn-health/shallow"
      expected_statuses: { start: 200, end: 299 }
  - timeout: 5s
    interval: 30s
    interval_jitter: 3s
    unhealthy_threshold: 2
    healthy_threshold: 2
    http_health_check:
      path: "/cdn-health/deep?check=db,storage"
      expected_statuses: { start: 200, end: 299 }

# outlier_detection for passive checks complements this
outlier_detection:
  consecutive_5xx: 5
  interval: 10s
  base_ejection_time: 30s
  max_ejection_percent: 40

Advanced Patterns and Troubleshooting When Intervals Go Wrong

When your 5s interval still feels too slow, don't just crank it to 1s — add passive health checking. Active checks are polling; passive is event-driven. We pair active 5s checks with Envoy's outlier detection, so if 5 consecutive requests 5xx within 10s, we eject the host in ~2-3 seconds without touching the active interval. It's the best of both worlds: you keep control-plane load at ~12k checks/minute per cluster instead of 60k, but your p99 detection for a truly dead origin drops from 15s to under 4s. That pattern alone cut our cross-region failover time by 62% without adding a single extra probe.

If you're on a consistent hash setup, this matters even more. A flapping node that bounces every 15 seconds will cause massive key redistribution — we've measured 18-23% cache churn on a 50-node ring when a single node flaps. Before you tighten intervals, make sure your Consistent Hash Ring Architecture for CDN: Minimizing Redistribution on Server Changes is using bounded-loads or virtual nodes to contain the blast radius. And if you're relying on IP Hash for CDN Session Persistence: Consistent User Routing Without Cookies, aggressive ejection is even riskier — you'll break session affinity for every user hashed to that node. In those architectures we actually *lengthen* active intervals to 10s and lean harder on passive ejection with a 30s ejection time.

Troubleshooting checklist when dashboards look off: First, graph health check QPS vs. origin CPU — if you see CPU spiking every 5s in perfect vertical lines, you've got no jitter. Second, check your false positive rate: if you're ejecting >2% of hosts per hour and 90% rejoin within 30s, your thresholds are too tight. Bump unhealthy_threshold from 2 to 3 or add 500ms to timeout. Third, watch for load balancer feedback loops — we've seen a case where a 2s interval generated 110k health checks/minute across 1,200 upstreams, which saturated the origin's accept queue and *caused* the 503s the checks were trying to detect. Dropping to 5s with jitter fixed it instantly and dropped check-related bandwidth from 480 Mbps to 95 Mbps.

Frequently Asked Questions