Load Balancer Observability Metrics: What to Monitor and Why for CDN Infrastructure

Back in 2019, we were handling a live sports final that pushed our edge to 1.2 Tbps. Every health check was green, CPU looked fine, and yet Twitter was on fire with complaints about buffering. Turns out our load balancer was silently shedding connections because its queue had filled up — and we weren’t monitoring queue depth at all. That night taught me you can’t trust a balancer’s status page; you’ve got to know which load balancer metrics actually tell you what users are feeling.

Why “All Backends Healthy” Doesn’t Mean Your CDN Is Healthy

I’ve seen this pattern more times than I’d like to admit: the balancer control plane reports 100% healthy nodes, but origin error rates are climbing and cache hit ratio is slipping. The problem is that most default health checks only tell you if a server responds to a TCP or HTTP ping, not if it’s serving production traffic well. A backend can pass a /healthz check in 10ms while simultaneously timing out on 30% of real requests because its thread pool is exhausted. If you’re only watching that binary healthy/unhealthy signal, you’re flying blind.

That’s why observability for CDN infrastructure has to start at the request level. You need per-backend and per-PoP breakdowns of request rate, HTTP status class distribution, and error rate normalized by traffic volume. A global 0.5% 5xx rate looks harmless until you slice it and realize one PoP in Frankfurt is returning 8% 502s for the last ten minutes. We learned to separate errors the balancer generates itself (like 503s from “no healthy upstream”) from errors it forwards from origins. The former tells you about GSLB Implementation Patterns: Global Server Load Balancing for Multi-Region CDN and capacity issues; the latter tells you about origin health. As defined in the HTTP Working Group Specifications, status codes like 502, 503, and 504 each signal a different failure mode, and grouping them together hides the root cause.

Don’t forget about request rate skew, either. When you’re running algorithms like Weighted Round-Robin for CDN: Proportional Traffic Distribution Across Heterogeneous Servers, you expect traffic to be proportional to weight. If a node with weight 10 is getting the same RPS as a node with weight 50, your weights are stale or your consistent hashing is flapping. I now alert on weight vs. actual RPS divergence, not just raw RPS.

# Alert when any backend's 5xx rate exceeds 1% over 5m, sliced by PoP and backend
sum by (pop, backend) (rate(haproxy_backend_http_responses_total{code=~"5.."}[5m]))
/
sum by (pop, backend) (rate(haproxy_backend_http_requests_total[5m])) > 0.01

# Separate balancer-generated 503s (no backend available) from origin 503s
sum(rate(loadbalancer_generated_503_total[2m])) by (pop)
# vs
sum(rate(backend_response_code_503_total[2m])) by (pop, backend)

Latency Percentiles and Saturation Signals That Predict Outages First

If error rates tell you what’s already broken, latency and saturation metrics tell you what’s about to break. Averages are useless here. I don’t care if your mean TTFB is 45ms if your p99 is 2.3 seconds — those 1% of users are the ones who churn. For CDN monitoring, you want at least p50, p95, and p99 for time-to-first-byte and for full downstream response time, segmented by cache status (HIT vs. MISS vs. REVALIDATE). A p95 that slowly creeps from 120ms to 400ms over a week is often the first sign of a degrading origin or a saturated balancer thread.

The second group we watch obsessively is saturation: active connections, connection queue depth, and connection reuse. Most modern balancers — NGINX, Envoy, HAProxy — can handle tens of thousands of concurrent connections, but every one has a limit on pending queue or worker connections. When we switched one large image PoP to Least-Connections Load Balancing for CDN: Dynamic Distribution Based on Server Load, we saw active connections become our most predictive signal. A backend with 2x the active connections of its peers wasn’t just “busier,” it was about to start queuing. And once queuing starts, latency percentiles explode before error rates move at all. The web.dev Articles on optimizing TTFB make this clear: queuing delay on the balancer directly inflates the user-perceived Largest Contentful Paint, even if your origin is fast.

We also track backend connection failures and retries. A single failed TCP connect followed by a successful retry is invisible to users but visible in your retry counter. When that counter jumps from 0.1% to 2%, it usually means a subnet or a rack is flapping. I’d rather get paged on that than wait for the 503s.

# Envoy / NGINX Prometheus alerts for latency and saturation
# p99 latency alert per route - fires before users notice widespread errors
histogram_quantile(0.99, sum by (le, pop) (rate(envoy_cluster_upstream_rq_duration_ms_bucket[5m]))) > 800

# Queue saturation - active vs. pending connections
# If pending queue > 0 for >2m, balancer can't drain fast enough
avg_over_time(haproxy_frontend_current_queue[2m]) > 0
or
avg_over_time(envoy_cluster_circuit_breakers_pending_open[2m]) > 0

# Connection reuse health - sudden drop indicates backend keepalive issues
rate(nginx_upstream_keepalive_cache_misses_total[5m]) / rate(nginx_upstream_requests_total[5m]) > 0.05

The Core Balancer Health Metrics You Should Put on Every Dashboard

After a decade of incidents, my team keeps a surprisingly short list of metrics that must be on every CDN load balancer dashboard. We split them into two layers: what the user experiences and what the balancer is doing internally. You need both. User-facing metrics like error rate and latency tell you if you have a problem; system metrics like health check failure rate and queue depth tell you where it is. Without that split, you’ll waste hours correlating logs during an outage.

We set service-level thresholds for each metric and tie them to runbooks. For instance, a health check failure rate above 10% doesn’t automatically drain a region, but it does trigger an automated check of the underlying network path and recent deploys. A backend connection pool exhaustion above 85% does trigger a weight reduction for that node. These aren’t arbitrary numbers — we tuned them by replaying past incidents and seeing which threshold would have paged us 5-10 minutes earlier without adding noise.

The table below is the starter set we use for any new PoP. You’ll adjust the numbers for your hardware and traffic mix, but don’t skip the metric entirely. If you’re not measuring it, you can’t alert on it, and if you can’t alert on it, you’ll find out from customers first.

Putting It Into Practice: Wiring Observability Into Your CDN Balancer

Knowing what to monitor doesn't help if your pipeline's dropping the data or your alerts fire 20 minutes too late. In a real CDN PoP, you've got milliseconds to decide if a backend is actually sick or just slow. We don't rely on default scrape intervals — we've seen a 15-second Prometheus scrape completely hide a 45-second origin retry storm. For edge balancers, we run a 5-second scrape for L7 metrics and push critical counters via OpenTelemetry to a local collector before they hit your central TSDB.

Start by tagging every metric with the dimensions you'll need to slice on during an incident: pop_id, balancer_instance, backend_pool, cache_status, and crucially, routing_key_hash if you're using session affinity. Without that last one, you can't tell if a latency spike is global or isolated to a single hot origin. This is where your load strategy matters — if you're running IP Hash for CDN Session Persistence: Consistent User Routing Without Cookies, a single /24 spiking can saturate one server while the dashboard average looks fine. We export upstream_hash_target as a label so we can group latency by the actual backend selected, not just the pool average.

Don't build alerts on averages. We alert on burn rate. A p95 of 180ms might be your normal at edge, but if your p95 to p99 gap suddenly goes from 40ms to 300ms, you've got head-of-line blocking or a saturated thread pool. Pair that with active connection and queue depth. Here's a production recording rule set we use on Envoy/NGINX-based edges to get actionable signals instead of noisy thresholds:

# Prometheus recording + alerting for CDN edge balancer
# Quantiles from Envoy's histogram or NGINX prometheus histogram

# 1. Pre-compute p95/p99 per pool per PoP (5m window)
- record: cdn:balancer_request_duration:p95_5m
  expr: histogram_quantile(0.95, sum by (le, pop_id, backend_pool) (rate(envoy_cluster_upstream_rq_time_bucket[2m])))

- record: cdn:balancer_request_duration:p99_5m
  expr: histogram_quantile(0.99, sum by (le, pop_id, backend_pool) (rate(envoy_cluster_upstream_rq_time_bucket[2m])))

# 2. Saturation signal: queue pressure vs. connection limit
- record: cdn:balancer_saturation_ratio
  expr: envoy_cluster_circuit_breakers_default_rq_pending{state="open"} / envoy_cluster_circuit_breakers_default_rq_pending{state="limit"}

# 3. Real alerts - burn rate, not static value
- alert: CDNHighTailLatency
  expr: cdn:balancer_request_duration:p99_5m > 450 and (cdn:balancer_request_duration:p99_5m - cdn:balancer_request_duration:p95_5m) > 250
  for: 3m
  labels: { severity: warning }
  annotations: { summary: "Tail latency explosion in {{ $labels.backend_pool }} at {{ $labels.pop_id }}" }

- alert: CDNBackendQueueSaturation
  expr: cdn:balancer_saturation_ratio > 0.8 or sum by (pop_id, backend_pool) (rate(envoy_cluster_upstream_rq_pending_overflow[1m])) > 0
  for: 2m
  labels: { severity: critical }

Advanced Patterns: Hunting Gray Failures and Distribution Skew

Once your dashboards are solid, you'll hit the failures that don't trip a health check. The classic is the "gray" backend — it passes your /healthz 200 OK in 5ms, but real asset fetches time out after 2 seconds because its disk is thrashing. If you're only doing shallow checks, your balancer will keep hammering it. That's why we run layered checks and mark a host unhealthy based on observed error rate, not just probe response. Check out the trade-offs in Health Check Design Patterns for CDN: Shallow, Deep, and Synthetic Checks — we've moved to a synthetic GET /__synthetic/1k-object.jpg that hits cache and disk on 20% of probes. If that synthetic p95 crosses 800ms, we eject the node for 30 seconds before it ever returns a 5xx to users.

The second killer is distribution skew, and it's brutal with stateful routing. Even with a good hash, you'll see 10-15% traffic variance between nodes under normal conditions. But when you add or remove an origin without a Consistent Hash Ring Architecture for CDN: Minimizing Redistribution on Server Changes, you can remap 40-60% of keys at once. We've watched p50 stay flat while one server's CPU hits 97% and its connection queue spikes to 800+ because it suddenly owned all the hot video segments. You won't catch this with pool-level metrics. You need per-backend request_rate, cpu_iowait, and ewma_latency on the same graph, plus a metric for key churn — hash_ring_reassignments_total — whenever the ring changes. If you see that counter jump and a single backend's p95 diverges by more than 2x from the pool median, you've got a hot shard, not a capacity problem.

Finally, watch your retries. Envoy and HAProxy will happily retry a 503 to another backend, which is great until a flapping origin causes a retry amplification loop. We track upstream_rq_retry and upstream_rq_retry_success as a ratio. If retries exceed 5% of total requests for more than 2 minutes, or if retry_success drops below 30%, we automatically circuit-break retries for that pool. A retry storm can turn one slow origin into a full PoP outage in under 60 seconds — your observability has to tell you to stop helping, not just that errors are climbing.

Frequently Asked Questions