Health Check Design Patterns for CDN: Shallow, Deep, and Synthetic Checks

Back in 2018, we pushed a new origin cluster live in Frankfurt and our dashboard stayed green for 20 minutes while users saw 500s. The load balancer was happily sending traffic to servers that returned 200 OK on /health but couldn't actually fetch anything from cache. That night taught me that a health check is only as good as what it actually verifies, and for CDN edge-to-origin traffic, getting it wrong means silent failures at scale.

Shallow Checks: Fast, Cheap, and Dangerously Optimistic

Shallow checks are the default most teams start with, and I get why. They're just a TCP connect or a simple HTTP GET to /healthz that returns a static 200. Your ADC or CDN director pings each origin every few seconds, and if it gets that 200 back within, say, 2 seconds, the node stays in the pool. It costs almost nothing, adds no load to your application, and catches the obvious stuff — box is down, Nginx crashed, network partition.

The problem is they don't tell you if the server can actually do its job. I've watched shallow checks pass while the origin's SSD was full, while Redis was wedged, and while the upstream CMS API was timing out after 30 seconds. The edge kept routing requests there because technically, the web server was alive. If you're running heterogeneous origins with different capacities, this gets worse. Your Weighted Round-Robin for CDN: Proportional Traffic Distribution Across Heterogeneous Servers logic will keep weighting that sick server as healthy, so you don't just get errors, you get skewed capacity planning. Shallow checks answer "is the process listening?" not "can it serve content?"

I still use shallow checks, but only as a first gate. Think of them as a liveness probe, not a readiness probe. For any-to-any PoP checks or for your outermost GSLB Implementation Patterns: Global Server Load Balancing for Multi-Region CDN layer, they're perfect because you want sub-second failover. You don't want GSLB waiting for a database query to decide a whole region is down. Keep them fast, keep the interval tight, and fail quickly.

# Nginx Plus / NGINX OSS with nginx_upstream_check_module
# Shallow check - just verifies TCP + HTTP 200 on a static endpoint

upstream cdn_origins {
    server 10.0.1.11:8080 max_fails=2 fail_timeout=10s;
    server 10.0.1.12:8080 max_fails=2 fail_timeout=10s;
    
    check interval=3000 rise=2 fall=3 timeout=2000 type=http;
    check_http_send "HEAD /healthz HTTP/1.0\r\n\r\n";
    check_http_expect_alive http_2xx;
}

# Origin side - keep it static, served from memory
location = /healthz {
    access_log off;
    add_header Content-Type text/plain;
    return 200 'ok';
    # Don't hit app, disk, or cache here
}

Deep Checks: Making Your Origin Prove It Can Work

Deep checks are where you force the origin to prove it can actually serve traffic. Instead of returning a static string, your /deep-health endpoint will touch the critical dependencies: read a canary object from your cache tier, run a SELECT 1 against the metadata DB, verify disk isn't read-only, and maybe fetch a 1KB object from your backing store. If any of those fail, you return 503 and the load balancer pulls you out. This is how you catch the Frankfurt scenario I mentioned — the server was up, but its cache volume was remounted read-only after a kernel panic.

You've got to be careful with how deep you go, though. I made this mistake in 2021 and built a deep check that hit three downstream services on every probe. At 200 origins checked every 5 seconds, we essentially DDoS'd our own database with health checks. A good deep check pattern samples dependencies, caches results for a few seconds, and has strict timeouts so the check itself doesn't hang. I've settled on a 1.5 second total budget for the deep endpoint — if it can't prove health that fast, it shouldn't be serving user traffic that expects < 200ms TTFB. As the HTTP Working Group Specifications remind us, semantics of status codes matter here — use 200 only for truly ready, 503 for degraded, and don't get creative with 204 or 429 for health.

Deep checks pair really well with Least-Connections Load Balancing for CDN: Dynamic Distribution Based on Server Load at the PoP level. When an origin starts to degrade — say its cache hit ratio drops and it's pulling more from origin storage — connection counts climb before it fully fails. A deep check that also reports load or latency lets your balancer make smarter decisions before the node hard-fails. We run deep checks on a slower interval than shallow checks, usually every 10-15 seconds, and we don't use them for GSLB decisions — only for PoP-local pool management.

// Go deep health handler - with timeouts and result caching
package health

import (
    "context"
    "net/http"
    "time"
)

var cachedStatus = struct {
    code int
    ts   time.Time
}{}

func DeepHandler(w http.ResponseWriter, r *http.Request) {
    // Cache result for 5s to avoid hammering dependencies
    if time.Since(cachedStatus.ts) < 5*time.Second {
        w.WriteHeader(cachedStatus.code)
        return
    }

    ctx, cancel := context.WithTimeout(r.Context(), 1500*time.Millisecond)
    defer cancel()

    checks := map[string]error{
        "cache_read":  checkCache(ctx),  // GET canary-key from Redis/Memcached
        "disk_rw":     checkDisk(ctx),   // write + delete temp file
        "metadata_db": checkDB(ctx),     // SELECT 1
    }

    for name, err := range checks {
        if err != nil {
            http.Error(w, "fail: "+name, http.StatusServiceUnavailable)
            cachedStatus = struct{ code int; ts time.Time }{503, time.Now()}
            return
        }
    }
    w.WriteHeader(http.StatusOK)
    w.Write([]byte("ready"))
    cachedStatus = struct{ code int; ts time.Time }{200, time.Now()}
}

Synthetic Checks: Simulating What Real Users Actually Do

Shallow asks "are you there?" Deep asks "are your dependencies up?" Synthetic asks "can a user actually watch the video or load the page?" That's the gap most CDN health monitoring misses. A synthetic check doesn't just hit an API endpoint — it performs a full transaction through the same path a user does: TLS handshake to the edge, request routing, cache lookup, origin fetch if miss, and header validation. We run synthetics that request a known /__synthetic/1kb.jpg object with cache-busting query params, then assert we get a 200, correct content-type, cache-control, and body checksum. If any of that fails, something in the delivery chain is broken even though every individual component says it's healthy.

This is where I've seen the biggest wins for CDN health monitoring. Last year a bad VCL deploy stripped CORS headers on edge but origin health was perfect — shallow and deep both green. Our synthetics caught it in 45 seconds because they validate response headers, not just status codes. As noted in web.dev Articles, user-perceived availability is about correct headers, caching, and rendering, not just TCP. We deploy synthetics from external vantage points and from inside the PoP. External tells you what users see, internal tells you if the problem is edge config or origin. They are heavier, so we run them less often — every 30 to 60 seconds — and we run them from a handful of probes rather than every load balancer.

The trade-off is cost and complexity. Synthetics can't run at the same frequency as shallow checks without burning bandwidth and polluting cache analytics. You'll want to tag synthetic requests so you can filter them from logs and billing, and you need to host canary objects that are small but realistic. Don't synthesize with a huge video segment either — you'll skew your origin egress numbers.

Practical Implementation: Building a Tiered Health Check Pipeline

You don't want every one of your 250+ edge PoPs hammering your origin database with deep checks every two seconds. That's how you DDoS yourself. The pattern that actually scales in production is tiered: run cheap shallow checks everywhere, expensive deep checks from a few aggregators, and synthetic checks from just a handful of canaries.

Here's what we run for a typical high-traffic property doing ~8M RPS globally: shallow HTTP checks (just GET /__healthz expecting a 200) from every edge every 2 seconds with a 500ms timeout. If an edge sees 3 consecutive failures, it doesn't immediately eject the origin — it flags it and leans on the regional health aggregator. That aggregator runs deep checks every 15 seconds against /__deepcheck, which actually pulls a 1KB canary object from cache, hits the database replica, and verifies TLS cert validity. You need that separation. If your consistent hash ring architecture re-maps 30% of your cache keys every time a single origin flaps, you've got a redistribution storm on top of an outage.

Synthetics sit at the top of the pyramid. We run 5 global canaries — not 100 — that perform a full TLS handshake, fetch a real product page, execute a single JS-rendered add-to-cart flow, and validate the TTFB is under 600ms. They run every 60 seconds. They're too heavy to be your primary failure detector, but they're the only ones that'll catch it when your origin returns 200 OK with a blank page because the SSR renderer crashed. Wire the tiers together with clear thresholds and don't forget the hold-down timer.

# tiered-healthcheck.yaml - used by our CDN origin director
origins:
  - name: origin-primary
    address: origin.example.com:443
    health_checks:
      shallow:
        endpoint: /__healthz
        interval: 2s
        timeout: 500ms
        expects: 200
        unhealthy_threshold: 3   # 6s to mark suspect
        healthy_threshold: 2

      deep:
        endpoint: /__deepcheck?canary=cache+db
        interval: 15s
        timeout: 2s
        expects: 200
        body_must_contain: '"db":"ok"'
        run_from: aggregators # only 3 regions, not all edges

      synthetic:
        script: ./synthetics/shopper-flow.js
        interval: 60s
        timeout: 8s
        slo: p95_ttfb < 600ms

    # Don't flap - require 30s stable before re-adding
    flap_damping:
      hold_down: 30s
      jitter: 10%

Advanced Patterns: Damping, Jitter, and When Checks Start Lying

The biggest operational headache isn't a hard-down origin. It's a flapping one. An origin that alternates between 200 and 500 every 10 seconds will destroy your cache efficiency if you let the load balancer react instantly. We've learned the hard way to add hysteresis. Once an origin is marked unhealthy, we enforce a 30-second hold-down even if the next probe is healthy, and we require 5 consecutive healthy deep checks (that's 75 seconds) before we slowly ramp traffic back with 10% weight increments. If you're doing IP hash for session persistence, a flap is even worse — you'll be bouncing users between origins and blowing up your session cache while you're at it.

You also need to solve the thundering herd. If all 250 edges probe at exactly 00, 02, 04 seconds, you get synchronized spikes on your origin. Add jitter. Our shallow probes use interval + random(-20%, +20%), so that 2-second check actually fires between 1.6s and 2.4s. For deep checks, we stagger aggregators by 5 seconds. And never, ever use the same health check endpoint as your readiness probe for autoscaling — we saw a team do that and their autoscaler spun up 40 new origins because the CDN's deep checks saturated the connection pool.

Finally, don't trust active probes alone. Pair them with passive observation. Your active checks might be green while 12% of real user traffic is getting 503s because of a routing rule that only applies to /api/*. That's where combining active vs passive health checks pays off. We export passive signals — origin 5xx rate, p95 origin latency, and connection reset counts — from every edge into the same decision engine. If passive 5xx exceeds 1% for 60 seconds, we force a deep check out-of-cycle, regardless of the schedule. It's saved us twice this year when health endpoints were healthy but the app was serving broken JSON to real users.

Frequently Asked Questions