Active vs Passive Health Checks for CDN: Probing vs Observation-Based Monitoring

Back in 2019, we had a PoP in Singapore that looked perfectly healthy to our synthetic probes while real users were getting 503s for 11 minutes. The active checker was hitting /healthz and getting a 200 OK, but the origin's content cache was corrupted and couldn’t serve actual assets. That night taught me you can’t rely on just one type of check — you need both probing and observation if you want honest HTTP Working Group Specifications compliant CDN monitoring that reflects what users experience.

Active Health Checks: Synthetic Probes That Don't Wait for Trouble

Active health checks are exactly what they sound like — your load balancer or CDN edge proactively sends synthetic requests to origins on a fixed interval, regardless of whether there’s user traffic. I think of them as a heartbeat monitor. Every few seconds, the system asks each server, "Are you still there? Can you handle a real request?" If the origin responds within the timeout window with the expected status code, it stays in the pool. If it fails a consecutive number of times, we pull it out before a single user request hits it.

What I've learned over the years is that the details of your probe matter more than the fact you're probing at all. A simple TCP connect check only tells you the port is open, not that Nginx or your app is actually serving content. For CDN origins, I always recommend an HTTP-level check against a lightweight but meaningful endpoint — something that touches the same dependencies as real content. Don't just check /ping that returns static text; check /healthz that verifies cache disk, upstream connectivity, and maybe a small object fetch. You’ll catch far more partial failures this way, especially when you’re running GSLB Implementation Patterns: Global Server Load Balancing for Multi-Region CDN where a region can appear up at Layer 4 but be broken at Layer 7.

The trade-off is overhead and configuration. If you’ve got 200 edge nodes each probing 10 origins every 2 seconds, that’s a lot of synthetic traffic you’re generating. You’ve got to tune intervals, timeouts, and thresholds carefully. Too aggressive and you’ll flap origins during a minor network blip; too lenient and you’ll be slower than passive detection. For heterogeneous pools where some servers are beefier than others, this pairs naturally with Weighted Round-Robin for CDN: Proportional Traffic Distribution Across Heterogeneous Servers — you don’t want to hammer a small origin with the same probe rate as your large ones.

# Active health check - NGINX Plus / CDN edge example
# Probes each origin every 3s, expects 200 within 1.5s
upstream cdn_origins {
    zone origins 64k;
    server origin1.cdn.example.net:443 max_fails=3 fail_timeout=10s;
    server origin2.cdn.example.net:443 max_fails=3 fail_timeout=10s;
    server origin3.cdn.example.net:443 max_fails=3 fail_timeout=10s;
}

server {
    location / {
        proxy_pass https://cdn_origins;
        health_check interval=3s
                     fails=3
                     passes=2
                     uri=/healthz
                     match=origin_ok;
    }
}

match origin_ok {
    status 200;
    header Content-Type = application/json;
    body ~ '"status":"ok"';
    timeout 1.5s;
}

Passive Health Checks: Letting Real User Traffic Do the Talking

Passive monitoring flips the model entirely. Instead of generating its own traffic, the edge watches live user requests and learns from their outcomes. If an origin starts returning 500s, 502s, or timeouts for real users, the load balancer quietly notices and starts reducing traffic to that node or ejecting it altogether. There’s no extra probe load, and you’re measuring what actually matters — the user experience, not a synthetic canary.

I’ve come to rely on passive checks for catching the weird, hard-to-simulate failures that active probes miss. Things like a single cache shard that’s corrupt, a TLS certificate chain that’s broken for certain clients, or an origin that’s technically up but incredibly slow. An active probe might still get its tiny /healthz response in 50ms, but users fetching a 2MB video chunk are seeing 8-second TTFBs. Passive observation catches that latency drift in real time. As described in web.dev Articles on real-user monitoring, observation-based signals are often closer to Core Web Vitals impact than any synthetic check. This is also where Least-Connections Load Balancing for CDN: Dynamic Distribution Based on Server Load benefits — when you combine live connection counts with passive error tracking, you stop sending traffic to struggling nodes automatically.

The catch? Passive monitoring is only as good as your traffic volume. If you have a low-traffic PoP or a newly added origin that isn’t getting many requests, you might not have enough data to make a confident decision. And by definition, passive means at least a few users have to fail before you react. We mitigate that by using outlier detection with sliding windows — eject an origin if, say, 5 out of 20 consecutive requests fail, but we don’t wait for hundreds of errors to pile up.

# Passive health check / outlier detection - Envoy proxy example
# Ejects origin based on real traffic errors, not synthetic probes
outlier_detection:
  consecutive_5xx: 5
  interval: 10s
  base_ejection_time: 30s
  max_ejection_percent: 50
  min_health_percent: 60
  failure_percentage_threshold: 25
  enforcing_consecutive_5xx: 100
  enforcing_failure_percentage: 100
  split_external_local_origin_errors: true

# Circuit breaker complements passive checks
circuit_breakers:
  thresholds:
    - max_connections: 10000
      max_pending_requests: 5000
      max_retries: 3
      max_requests: 8000

Probing vs. Observation: Which Fails Faster and Costs Less?

So which one should you use? Honestly, it isn’t an either-or decision. In my production setups, we run both simultaneously and let them cover each other’s blind spots. Active checks give you fast, proactive detection even with zero traffic — perfect for cold origins, disaster recovery regions, or that 3 AM period when traffic is low. Passive checks give you truth — they tell you when real users are hurting, even if your health endpoint says everything’s fine.

The failure modes are completely different. Active checks can give you false positives if your probe endpoint is misconfigured or if there’s transient jitter between the edge and origin. We once had an origin get marked down globally because our health check URI was accidentally blocked by a WAF rule — the servers were fine, but our probes weren’t. Passive checks, on the other hand, can give you false negatives under low load; a failing origin that only receives 1 request per minute might not trip a 50% error threshold for a long time. That’s why combining them matters — active catches the low-traffic failure quickly, passive catches the subtle content-level failure accurately.

When I’m designing a new tier for clients, I usually start with this comparison framework. It helps teams decide where to invest tuning effort based on their traffic patterns and tolerance for synthetic load. There’s no single right answer, but you should be explicit about what you’re optimizing for — detection speed, accuracy, or overhead.

Hybrid in Practice: Wiring Active Probes and Passive Observation Together

Look, you don't pick one. Every production CDN I've tuned in the last five years runs both, and the trick is letting each do what it's good at without stepping on the other's toes. Active checks are your early-warning radar — they catch a dead origin in 10-15 seconds before a single user request gets burned. We typically run them every 5 seconds with a 2-second timeout, requiring 3 consecutive failures to mark down and 2 consecutive successes to bring it back. That dampens flapping when an origin is just GC-pausing. Passive checks, on the other hand, are your reality check. They don't care if /healthz returns 200; they care if real GETs for actual objects are returning 503s or taking 2.4 seconds on a p95 that should be 120ms.

The pattern that works best is active for liveness, passive for quality. Use a shallow active probe for fast failure detection, then layer a passive outlier detector that watches a sliding 30-second window of real traffic. If the 5xx rate exceeds 5% or p95 latency crosses 800ms for that origin, you eject it for 30-45 seconds — even if the active probe still says "healthy." This saves you from the classic case where the health endpoint is served from cache or a lightweight thread while the application workers behind it are saturated. If you're designing your probes, read up on Health Check Design Patterns for CDN: Shallow, Deep, and Synthetic Checks — a deep check that hits downstream dependencies will catch what a shallow TCP check never will.

Here's how that hybrid looks in an Envoy-based edge — which is what most modern CDNs are converging on — combining active health checking with passive outlier detection:

clusters:
- name: cdn_origin_cluster
  connect_timeout: 2s
  lb_policy: RING_HASH # or MAGLEV for better distribution
  health_checks:
  - timeout: 2s
    interval: 5s
    unhealthy_threshold: 3
    healthy_threshold: 2
    http_health_check:
      path: /healthz/deep  # hits app + cache + DB read-replica
      expected_statuses: { start: 200, end: 299 }
  outlier_detection:
    consecutive_5xx: 5
    interval: 10s
    base_ejection_time: 30s
    max_ejection_percent: 50
    # Passive: eject if p95 latency or error rate spikes
    enforcing_consecutive_5xx: 100
    enforcing_success_rate: 100
    success_rate_minimum_hosts: 5
    success_rate_stdev_factor: 1900 # ~1.9 stddev = eject outliers
  circuit_breakers:
    thresholds:
    - max_retries: 3
      max_pending_requests: 10000
      max_requests: 5000

Advanced Patterns and Troubleshooting: When Health Checks Lie to You

Health checks fail in the same three ways every time, and they'll make you page at 3am if you don't plan for them. First is the flap storm. An origin hovering at 50% packet loss will bounce between healthy and unhealthy every 15 seconds, and if you're using a naive Consistent Hash Ring Architecture for CDN: Minimizing Redistribution on Server Changes, each flap reshuffles 10-20% of your keys and nukes your cache hit ratio from 92% down to 68%. We've fixed this by adding jitter (randomize intervals by +/- 1s), doubling the healthy_threshold to 4 on flapping hosts, and implementing a slow-start. When an origin returns, don't slam it with 100% traffic — ramp it over 30-60 seconds while it warms cache. Envoy's slow_start_window or NGINX's slow_start=30s does this automatically.

Second is the "healthy but useless" origin. Your active probe returns 200, but real users get 503s because your passive check never accounted for session affinity. This is brutal if you're using IP Hash for CDN Session Persistence: Consistent User Routing Without Cookies. If you eject an origin that holds 30k persistent connections based on IP hash, those users don't just fail over cleanly — they get re-hashed, lose session state, and hammer the origin that just came back. Troubleshoot this by making your passive detector routing-aware: don't eject on global error rate, eject based on per-origin success rate *compared to the cluster mean*. If one origin's 5xx rate is 2.5 standard deviations above the cluster average over a 30s window, it's an outlier — eject just that one for 45s and let IP hash naturally rebalance.

Finally, watch for health-check amplification. At scale, 2,000 edge PoPs probing 100 origins every 5 seconds is 40k RPS of synthetic traffic you didn't budget for. I've seen health checks DDoS the origin they were meant to protect. Solution: aggregate checks at the regional layer. Have 3-5 designated probers per region do the active checks and gossip the result via your control plane, rather than having every edge node probe independently. Keep the per-edge passive observation local and cheap — it's just counting counters on the request path, zero extra traffic.

Frequently Asked Questions