HTTP Health Check Headers for CDN: Custom Probes for Application-Level Verification

Back in 2019, I watched a perfectly healthy-looking origin cluster pass every TCP check our CDN threw at it — while serving 500 errors to every real user for twelve minutes. The load balancer thought everything was fine because the socket opened. The application behind it was dead, and we didn’t know until customers told us. That’s the gap HTTP health check headers and custom probes are built to close.

Why TCP Checks Alone Can't Catch Application Failures

Most default CDN health checks are embarrassingly simple. They open a TCP connection to port 80 or 443, maybe send a bare-bones HEAD / request, and if they get a SYN-ACK or any 2xx back, they mark the node as healthy. It’s fast and cheap, but it tells you almost nothing about whether your application can actually serve traffic. I’ve seen database connection pools exhaust, disk fills that break image rendering, and bad deploys where the app returns a 200 OK with a body that says “maintenance” — all while the TCP probe stayed green.

An effective HTTP health check moves the verification up the stack. Instead of asking “is the machine listening?” you’re asking “can this specific application serve a realistic request and return the expected headers and payload?” That means your CDN edge isn’t just checking connectivity; it’s performing an application check. When you combine that with intelligent traffic distribution like Least-Connections Load Balancing for CDN: Dynamic Distribution Based on Server Load, you prevent the classic failure mode where the least-loaded server is actually the most broken one and keeps getting rewarded with more traffic.

For a global footprint, this gets even more critical. A simple ping from a single edge PoP won’t catch a regional cache or database replica that’s lagging. That’s why teams running multi-region setups pair these application-level probes with GSLB Implementation Patterns: Global Server Load Balancing for Multi-Region CDN. The pattern’s the same: don’t trust the lower layers to tell you what’s happening at the top.

# Naive TCP vs. Application-Aware HTTP Health Check
# This probe will PASS even if the app is broken

# --- Bad: Only checks if port is open ---
probe tcp_only {
  .url = "/";
  .expected_response = 200; # Any 200 passes, even error pages
}

# --- Good: Verifies application logic via headers and body ---
probe app_aware {
  .url = "/healthz"
  .request =
    "GET /healthz HTTP/1.1"
    "Host: origin.example.com"
    "User-Agent: CDN-HealthChecker/2.0"
    "X-Probe-Type: cdn-health"
    "X-Health-Check: deep"
    "Cache-Control: no-cache";
  .expected_response = 200;
  .expected_header = "X-App-Status: healthy";
  .expected_body_regex = "\"db\":\s*\"ok\"";
  .timeout = 3s;
}

Building Custom Probes That Verify More Than a Status Code

Custom probes are where you stop treating health checks as a checkbox and start using them as a diagnostic tool. The trick isn’t to hit your homepage — it’s to build a dedicated, lightweight endpoint that exercises the critical path without creating side effects. My go-to is a /healthz or /cdn-status endpoint that checks downstream dependencies (primary DB, Redis, object storage) and returns a structured response with CDN health headers. You don’t want it doing heavy work, but you do want it to fail fast if anything the user journey depends on is down.

The headers you send *to* the origin matter just as much as what you expect back. I always send a distinct set for probes so the origin can handle them differently: a custom `X-Probe-Type`, a `User-Agent` that’s easy to filter in logs, and `Cache-Control: no-cache, no-store` plus `Pragma: no-cache` to bypass any intermediate cache. On some stacks I also add `X-CDN-Edge: fra-12` so the origin knows which PoP is checking it. That’s saved us hours of debugging when a single edge had routing issues that didn’t show up elsewhere. Your origin should respond with its own headers — I like `X-App-Status`, `X-App-Version`, and `X-Dependency-Status` — so the CDN can make a decision without parsing the body. The HTTP Working Group Specifications at https://httpwg.org/specs/ are the right reference for how intermediaries should handle these cache-control and custom header semantics, and articles on https://web.dev/articles/ do a solid job explaining why you don’t want health checks cached.

You should also think about what “healthy” means for different kinds of traffic. If you’re running Weighted Round-Robin for CDN: Proportional Traffic Distribution Across Heterogeneous Servers, a heavy origin that’s degraded but not dead might still pass a simple check and soak up its weighted share. A better custom probe can return a degraded state — say, `X-App-Status: degraded` with a 200 — and your CDN logic can temporarily reduce that node’s weight or drain it instead of hard-failing it. It’s a more graceful way to handle those in-between states that aren’t fully up or fully down.

# Example: CDN origin probe config (Varnish / Fastly VCL style)
backend origin_us_east {
  .host = "10.20.1.50";
  .port = "8080";
  .probe = {
    .url = "/cdn-status";
    .request =
      "GET /cdn-status HTTP/1.1"
      "\r\nHost: www.example.com"
      "\r\nUser-Agent: CDN-Probe-Edge/1.0"
      "\r\nX-Health-Check: deep"
      "\r\nX-Probe-ID: ${EDGE_ID}"
      "\r\nCache-Control: no-cache"
      "\r\nConnection: close";
    .interval = 5s;
    .timeout = 2s;
    .window = 5;      # Sliding window for evaluation
    .threshold = 3;   # 3/5 must pass to be healthy
    .expected_response = 200;
    # Require custom header; fail if it's missing or != healthy
    .expected_response_headers = "X-App-Status: healthy";
  }
}

# Origin (Node.js/Express) handler example
app.get('/cdn-status', (req, res) => {
  if (req.get('X-Health-Check') !== 'deep') {
    return res.status(400).end();
  }
  const checks = {
    db: db.pingOk() ? 'ok' : 'fail',
    cache: redis.pingOk() ? 'ok' : 'fail'
  };
  const healthy = Object.values(checks).every(v => v === 'ok');
  res.set({
    'X-App-Status': healthy ? 'healthy' : 'unhealthy',
    'X-App-Version': process.env.GIT_SHA,
    'Cache-Control': 'no-store, private'
  });
  res.status(healthy ? 200 : 503).json(checks);
});

Tuning Probe Headers, Timing, and Failure Thresholds

Getting the probe logic right is only half the job — you’ve got to tune how often it runs and how quickly you react. Too aggressive and you’ll flap origins during a minor GC pause; too relaxed and you’ll keep serving errors for a minute before the CDN pulls the node. I’ve learned the hard way that headers help you control that noise. Sending `Cache-Control: no-cache` and ensuring your origin responds with `Cache-Control: no-store, private, max-age=0` guarantees every probe actually hits the app. Without those, I’ve seen CDNs cache a healthy 200 response for 30 seconds and completely mask a failure that happened five seconds later.

You don’t want a single timeout to mark a whole region as down, so use a window/threshold model. Most production setups I run don’t make a decision on one request — they look at 3 successes out of 5 attempts, or 2 consecutive failures to go unhealthy and 3 consecutive successes to come back. That hysteresis prevents flapping when an origin is under brief pressure. You’ll also want different intervals for edge vs. origin health: edges can probe every 5 seconds with a 2-second timeout, while cross-region GSLB probes might be 10-15 seconds to avoid inflating global traffic.

Putting Custom Probes Into Production: Edge and Origin Configuration

Knowing what headers to send is one thing — making your origin and edge actually respect them is where most teams stumble. You don't want health checks hitting your normal application path and polluting cache or triggering rate limits. The cleanest approach we've seen is a dedicated probe endpoint that only responds when it sees your custom headers, and returns a hard 404 or 444 to everything else. That keeps scanners and curious users out while giving your CDN a reliable signal.

On the origin side, say Nginx, you'll want to gate the endpoint with the probe headers and run your application-level checks behind it. Don't just return 200 OK — have the handler verify DB connectivity, Redis latency under 50ms, and that your last deployment asset manifest exists. On the CDN side, configure the monitor to inject those headers, expect a specific body string, and fail closed if anything is off. We typically run probes every 5 seconds with a 2-second timeout, marking an origin down after 3 consecutive failures and bringing it back after 2 consecutive successes. That gives you a 15-second detection window without flapping on a single slow GC pause.

If you're using consistent hashing for your origin shield, this matters even more. A flapping origin that gets ejected and re-added every 30 seconds will thrash your cache and spike origin load. Pair your probe tuning with your architecture — if you're still weighing routing strategies, it's worth revisiting how IP Hash for CDN Session Persistence: Consistent User Routing Without Cookies and Consistent Hash Ring Architecture for CDN: Minimizing Redistribution on Server Changes affect failure domains. You don't want a health check design that re-hashes 40% of your keys every time one node hiccups.

# Nginx origin: gated deep health check
# Only responds to CDN probes with correct headers

location /cdn-health/deep {
    # Drop any request without our secret probe headers
    if ($http_x_cdn_probe != "verify-v2") { return 404; }
    if ($http_x_probe_token != "k8s-prod-cdn-9f3a") { return 404; }

    # Don't log probes, don't cache them
    access_log off;
    add_header Cache-Control "no-store, private" always;
    proxy_no_cache 1;

    # Proxy to your app's internal health handler (e.g., Node/Python/Go)
    # That handler should check DB, cache, and disk and return JSON
    proxy_pass http://127.0.0.1:3000/internal/health/deep;
    proxy_connect_timeout 1.5s;
    proxy_read_timeout 2s;
}

# CDN Monitor Config (example: Varnish / Fastly / CloudFront style)
# Interval: 5s | Timeout: 2s | Window: 10s
# Unhealthy threshold: 3 failures | Healthy threshold: 2 successes
# Expected: HTTP 200 + Body contains '"status":"healthy"' + Header X-App-Version matches deploy

Advanced Patterns and Troubleshooting When Probes Lie

Once your basic probe is solid, you can start using headers to do smarter work. One pattern we love is header-driven depth selection. Send X-Probe-Depth: shallow from your edge POPs for the frequent 5-second check — just verify the app process is up and responding in under 100ms. Then send X-Probe-Depth: deep from a single global checker every 30 seconds that hits downstream dependencies. If shallow passes but deep fails, keep the node in rotation but fire a PagerDuty warning. This prevents a slow downstream Redis replica in us-east-1 from taking your entire origin pool offline. For a full breakdown of when to use each, see Health Check Design Patterns for CDN: Shallow, Deep, and Synthetic Checks.

Another powerful trick is using probe headers for canary and version verification. Have your deployment inject X-Expected-Version: v2.4.1-b7a9e2 into the CDN probe config right after a rolling deploy. Your health handler should compare that header to the running build hash and return 503 if they don't match. The CDN will naturally drain traffic from stale nodes without any manual intervention. We've used this to catch the classic "Nginx reloaded but app didn't restart" failure that a naive 200 OK would completely miss. Just be careful to roll the probe config and the code together — otherwise you'll mark every node unhealthy for 2 minutes.

When probes are flaky, don't immediately blame the network. 90% of false positives we've debugged come down to three things: origin Keep-Alive timeouts shorter than the probe interval (set origin keepalive to at least 30s if you probe every 5s), probe requests getting rate-limited by a WAF rule that doesn't exclude your probe User-Agent, and missing Vary handling that causes a health response to get cached and served as a user response. If your probe starts returning 200 but latency jumps from 20ms to 1.8s, that's not a pass — tune your CDN to fail on p95 latency thresholds too. Add X-Probe-ID: <pop>-<timestamp> and log it origin-side so you can correlate exactly which POP saw the failure.

Frequently Asked Questions