Origin Health Monitoring Patterns: Active Probes, Passive Checks, and Hybrid Approaches

At 2:13 AM on Black Friday, our primary origin pool started returning 500s. The CDN edge kept hammering it for another 45 seconds because our health checks were set to a lazy 30-second interval. By the time we marked that origin as down, we'd already served thousands of errors that should've gone to the secondary. That night rewired how I think about origin health monitoring entirely.

Active Probes: A Predictable Heartbeat You Control

Active probes are synthetic requests your CDN sends on a fixed schedule, regardless of whether real user traffic is flowing. Think of them as a heartbeat monitor. You decide the endpoint, the interval, the timeout, and what counts as healthy. Most teams point them at a lightweight /healthz or /__status endpoint that checks downstream dependencies like cache, DB, and disk, then returns a 200 only if everything's truly OK. This is the cleanest signal you can get, because you're not mixing health data with user experience.

I've run active checks for origins that serve barely any traffic overnight, and they're invaluable there. When a low-traffic origin fails at 3 AM, you don't want to discover it from the first real user after 20 minutes of silence. A 5-second probe catches it immediately. The trade-off is cost and fidelity. Too frequent and you're adding thousands of requests per minute across a fleet of edges. Too infrequent and you're slow to detect failure. And synthetic checks can lie — I've seen origins return 200 on /healthz while the app server thread pool was completely saturated for real asset paths.

The trick is to make probes realistic without being expensive. Don't just check TCP connect; send a real HTTP request with Host header, TLS, and follow redirects. Validate status code *and* response body, and enforce a strict timeout that's tighter than your origin read timeout. If you use connection reuse for probes, isolate them — you don't want a probe borrowing a poisoned connection from the Origin Connection Pooling Optimization: Reducing TCP Overhead at CDN Scale pool and giving you a false pass. We also add jitter to probe intervals so a thousand edges don't thundering-herd the origin at the same second.

# Active health check config (Envoy / CDN edge style)
health_checks:
  - timeout: 2s
    interval: 5s
    unhealthy_threshold: 2
    healthy_threshold: 3
    interval_jitter: 0.5s
    http_health_check:
      path: "/__origin_health"
      expected_statuses:
        - start: 200
          end: 299
      expected_response_body: '"status":"ok"'
      request_headers_to_add:
        - header:
            key: "User-Agent"
            value: "cdn-health-prober/1.0"
# Fail fast, recover cautiously. 2 failures = out, 3 successes = back in.

Passive Monitoring: Letting Real Traffic Report Origin Status

Passive monitoring doesn't send anything extra. It watches real user requests as they pass through and infers origin status from the results. If an origin starts throwing 5xx errors, timeouts, or TCP resets, the edge increments a failure counter. Cross a threshold, and that origin is ejected from the pool without ever needing a synthetic probe. This is how you catch the failures active checks miss — like when /healthz is fine but /api/checkout is melting down under load.

Where passive checks shine is speed and realism. You're measuring what users actually experience, not an idealized health endpoint. I once debugged a case where active probes reported 100% healthy while P95 origin latency had crept to 4.2 seconds. Users were abandoning pages, but our health system thought everything was fine. A passive monitor tracking latency and 5xx rate would've ejected that origin in seconds. The implementation details matter here — you need to separate origin-induced errors (502, 503, 504, connection timeout) from client errors (4xx) or you'll flap an origin for no reason. And you'll want the guidance in the HTTP Working Group Specifications for interpreting status codes as you define what counts as unhealthy.

But passive monitoring has a cold-start problem. No traffic means no signal. It also needs careful tuning to avoid overreacting to a brief spike. We use a sliding window with outlier detection — for example, eject if more than 30% of requests in the last 20 requests fail, or if 5 consecutive requests timeout. Pair that with an ejection duration and you avoid flapping. If you haven't set up your failure handling, read Origin Retry Strategies for CDN Resilience: Backoff, Jitter, and Budget Limits before you enable passive ejection, because a badly tuned passive monitor can amplify retries and make an outage worse. The best setups also feed this signal into Origin Request Hedging: Sending Parallel Requests to Cut Tail Latency logic so slow origins get deprioritized before they're fully removed.

// Passive health tracking (pseudo-logic at edge)
const WINDOW_SIZE = 20;
const FAILURE_THRESHOLD = 0.30; // 30% failures
const CONSECUTIVE_TIMEOUTS = 5;

function onOriginResponse(origin, result) {
  origin.window.push(result); // { status: 503, latencyMs: 2100, error: 'timeout' }
  if (origin.window.length > WINDOW_SIZE) origin.window.shift();

  const failures = origin.window.filter(r => 
    r.status >= 500 || r.error === 'timeout' || r.error === 'reset'
  ).length;

  const consecutiveTimeouts = countTrailing(origin.window, r => r.error === 'timeout');

  if (failures / origin.window.length > FAILURE_THRESHOLD || 
      consecutiveTimeouts >= CONSECUTIVE_TIMEOUTS) {
    ejectOrigin(origin, { duration: '30s', reason: 'passive_failure_rate' });
  }
}

Choosing Your Signal: Overhead, Detection Time, and Accuracy

Neither approach is universally better. Active probes give you consistency during low traffic and a clear contract with origin owners ("return 200 on this path"), but they add overhead and can miss path-specific failures. Passive checks give you ground truth from real traffic with zero extra requests, but they're blind when traffic is sparse and can be noisy if you don't filter correctly. Most mature CDNs I've worked with don't choose — they use both.

The decision often comes down to your traffic shape and page performance goals. For high-traffic origins serving static assets, passive monitoring alone catches 90% of issues within a second or two. For low-traffic API origins or cold secondary pools, you can't rely on passive — you need active to maintain readiness. And if you're optimizing for Core Web Vitals, remember that health detection time directly impacts LCP and INP for affected users. Resources on web.dev Articles show how tail latency from a sick origin can blow your latency budgets.

Here's how the two patterns compare in production environments I've measured. Numbers are for a typical mid-size CDN PoP with ~5k req/s per origin pool.

From Theory to Config: Building a Hybrid Check That Actually Works

Here’s how we typically wire it up in production. You don’t want to choose between active and passive — you want both voting on the same origin pool. I like to run active probes on a tight 5-second interval with a 2-second timeout, expecting a 200 from a dedicated /healthz endpoint that actually hits your cache and DB, not just returns 200 OK from nginx. In parallel, every edge PoP keeps a rolling passive window — last 20 requests or 30 seconds, whichever comes first. If 30% of those real requests hit a 5xx, timeout, or connection error, that’s a failure vote even if the active probe is still green.

The failover logic is where most teams get burned. Don’t mark an origin down on a single blip. We use 3 consecutive active failures to pull it, and require 2 consecutive successes to bring it back — that dampens flapping when an origin is hovering around 300-400ms. And your health check timeouts have to match your real traffic timeouts, or you’ll create split-brain decisions. If your edge is configured to give up on an origin after a 3-second connect timeout, but your probe waits 5 seconds, the probe will think the origin is healthy while users are already failing. Nail down that relationship first — our Origin Timeout Configuration Guide: Connect, Read, and Total Timeout Tuning breaks down how to align connect, read, and total timeouts so your monitors and your user traffic agree on what “down” means.

One more thing you’ll thank yourself for later: make your probe do what real traffic does. If your origins enforce HTTPS, your health check must validate the certificate chain, SNI, and hostname just like a client request. We’ve seen probes passing on self-signed certs while real user fetches were failing TLS verification at the edge. Set verify_ssl: true and pin the expected hostname. It’s covered in detail in the Origin SSL Verification at CDN Edges: Preventing Man-in-the-Middle Attacks guide, but for health checks it’s non-negotiable — a probe that ignores TLS isn’t probing.

{
  "origin_group": "api-prod-us-east",
  "origins": ["10.0.1.10", "10.0.1.11", "10.0.1.12"],
  "health_check": {
    "active": {
      "path": "/healthz?deep=1",
      "interval_ms": 5000,
      "timeout_ms": 2000,
      "healthy_threshold": 2,
      "unhealthy_threshold": 3,
      "expected_status": [200],
      "expected_body": "ok",
      "verify_ssl": true,
      "host_header": "api.example.com"
    },
    "passive": {
      "enabled": true,
      "window": "20_requests_or_30s",
      "error_threshold_percent": 30,
      "count_timeouts_as_errors": true,
      "ignore_codes": [429, 503],
      "cooldown_ms": 15000
    },
    "failover": "quorum_global_with_local_fast_fail"
  }
}

When Health Checks Lie: Flapping, Gray Failures, and Edge Disagreements

If you’ve run health checks at scale, you know the worst failures aren’t hard-down — they’re gray. Your origin returns 200 to the probe in 40ms, but real requests with a large header or POST body time out after 4 seconds because a backend worker pool is saturated. That’s why we always tell teams to ignore 429s and 503s with Retry-After in passive checks for 10-15 seconds — they’re often intentional backpressure, not a dead origin. Conversely, don’t ignore a steady climb in p95 latency; if your passive monitor sees read TTFB jump from 120ms to 850ms for 60 seconds straight, you should shed load before you hit hard errors. A good hybrid setup treats latency degradation as a soft failure.

Flapping is the other classic. You’ll see an origin marked down in IAD, up in FRA, down again 20 seconds later. That usually means you’ve got PoP-local health state without a global quorum. At a thousand edges, each PoP making isolated decisions is chaos. We use local fast-fail (an edge can route around a bad origin in <50ms based on its own passive window) but require a global quorum — say 60% of PoPs reporting 3 consecutive active failures — before we drain the origin from DNS or Anycast entirely. It adds 8-12 seconds to global detection time, but it prevents one noisy PoP with packet loss from taking down an origin for everyone.

And keep an eye on the origin side too. If you’re rotating origin IPs frequently or they land on a blocklist, your SYN packets might be getting dropped long before they hit your app — your health check will just see “timeout” with zero clue why. We’ve debugged cases where an origin’s IP reputation tanked due to a compromised neighbor on the same /24, and every edge probe started failing connect in 2 seconds flat. If you’re seeing 100% connect timeouts but the box looks fine locally, check your Origin IP Reputation Management: Keeping Your CDN Source IPs Clean status and whether your security groups are rate-limiting the CDN’s probe ranges. Don’t let a firewall rule make you think your app is down.

Frequently Asked Questions