CDN Degradation Detection: Finding Partial Outages Before They Become Total Failures

Last summer I got paged at 2:13 AM for a "non-incident." Our global uptime dashboard was a perfect wall of green, 100% availability across all pops, yet customer support was drowning in complaints about slow image loads in Southeast Asia. It wasn't a total failure, so none of our alerts fired. That night we spent four hours chasing what turned out to be a 12% packet loss on a single transit provider in Singapore — a classic partial outage that our binary up-or-down monitoring completely ignored. If you've run a CDN long enough, you've lived this story too.

Why Binary Health Checks Miss the Failures That Hurt Most

For years we treated CDN monitoring like a light switch: either the edge is up or it's down. A simple HTTP 200 from a synthetic probe in ten regions and we called it healthy. That approach works for catastrophic failures — a PoP going offline or a bad config push that 500s everywhere — but it falls apart for the messy, creeping failures that make up most real-world pain. A partial outage rarely announces itself. It's a 300ms bump in TTFB in only one region, or a cache hit ratio that slips from 96% to 88% over an hour.

I've learned that customers feel degradation long before our status page does. They don't care if our status checks pass; they care that their LCP scores just tanked and their conversion rate is dipping. The problem is that traditional threshold alerts are too noisy or too numb. Set them too tight and you get alert fatigue, set them too loose and you miss the early warning. What we needed was a way to define degradation distinct from failure. That's where thinking in terms of SRE Practices for CDN: Applying Site Reliability Engineering to Content Delivery really changed how my team instrumented things — we stopped asking "is it up?" and started asking "is it performing as users expect?"

The HTTP specs themselves give us clues about what to measure beyond status codes. As described in the HTTP Working Group Specifications, signals like stale cache revalidation, 103 Early Hints timing, and retry behavior tell you far more about user experience than a 200 OK. I now treat any consistent shift in those secondary signals as a potential incident, even if no hard errors are logged. You can't fix what you don't define, so we had to codify what "degraded" actually means for our platform.

# PromQL: Detect regional TTFB degradation before it breaches SLO
# Compares p95 TTFB per PoP against a 7-day rolling baseline
(
  histogram_quantile(0.95, sum by (pop, le) (rate(cdn_edge_ttfb_bucket[5m])))
  /
  avg_over_time(
    histogram_quantile(0.95, sum by (pop, le) (rate(cdn_edge_ttfb_bucket[5m])))[7d:5m]
  )
) > 1.5
and
  sum by (pop) (rate(cdn_edge_requests_total[5m])) > 100
# Fires only if p95 is 50% higher than baseline AND pop has traffic

Designing Early Warning Signals That Don't Cry Wolf

The trick isn't collecting more metrics — we're already drowning in them — it's combining weak signals into a strong one. A single metric blip means nothing, but three correlated shifts in the same PoP? That's almost always real. Back in 2019, we had a cache tier in Frankfurt where the SSDs started failing gradually. No single drive failed outright, so RAID stayed green, but p99 origin fetch latency crept up 40ms per day for a week. Individually, none of our alerts cared. Together, they were screaming.

We've moved to a composite scoring model for CDN monitoring. Instead of alerting on raw thresholds, we evaluate the rate of change and the breadth of impact. For early warning, I look for three things simultaneously: latency distribution widening, cache efficiency dropping, and origin pressure rising — all in the same geographic shard. If you see that trio, you don't have noise, you have a partial outage in progress. The guidance in web.dev Articles on Core Web Vitals is useful here too; a shift in LCP or CLS at the edge often shows up in our RUM data 10-15 minutes before our synthetic probes catch it.

This approach ties directly into how we manage reliability trade-offs. You can't alert on every wobble without burning out your on-call. We use CDN Error Budget Management: Balancing Reliability with Feature Velocity to decide which degradations are worth paging for. A 20% TTFB bump in a low-traffic PoP might just be a ticket for business hours, but the same bump in our top three traffic regions burns error budget fast and deserves an immediate page. That context prevents the "alert fatigue" death spiral that kills most degradation detection efforts.

#!/usr/bin/env python3
# Simple degradation detector: z-score on cache hit ratio per PoP
import numpy as np

def is_degraded(current_hit_ratio, history, z_threshold=3.0):
    """
    history: list of hit ratios for same PoP, same hour, last 14 days
    Returns True if current value is anomalously low
    """
    mean = np.mean(history)
    std = np.std(history)
    if std == 0:
        return False
    z_score = (current_hit_ratio - mean) / std
    # Negative z-score means worse than average
    if z_score < -z_threshold:
        print(f"Degraded: {current_hit_ratio:.2%} vs baseline {mean:.2%} (z={z_score:.2f})")
        return True
    return False

# Example: hit ratio dropped to 89% vs 96.5% baseline
history_baseline = [0.965, 0.962, 0.968, 0.964, 0.967, 0.963, 0.966] * 2
is_degraded(0.89, history_baseline)

What Actionable Degradation Thresholds Actually Look Like

Teams often ask me for the "right" numbers to alert on. There's no universal answer, but I can share what we've settled on after a lot of painful tuning. The key is to base thresholds on your own baseline distributions, not static industry numbers. A 150ms TTFB might be fantastic for a PoP in Johannesburg fetching from an origin in Virginia, but terrible for a PoP in Ashburn doing the same. Your thresholds have to be relative and regionally aware.

We've also learned to automate the boring parts of triage so humans can focus on judgment. When a degradation signal fires, our system now automatically correlates it with recent deploys, transit provider health, and origin latency — work that used to take an engineer 30 minutes at 2 AM. Following principles from CDN Toil Reduction Through Automation: Eliminating Repetitive Operations Work cut our mean time to acknowledge for partial outages from 42 minutes to under 6. The automation doesn't decide if it's an incident, it just hands the on-call a complete picture so they can.

Don't aim for perfection on day one. Start with four or five high-signal metrics and get their baselines right. It's better to have reliable early warning on TTFB and cache hit ratio than flaky coverage on twenty signals. The table below shows the starting thresholds we use in production — adapt them to your traffic and SLOs, but treat the deltas as more important than the absolute values.

From Theory to Production: Wiring Degradation Detection Into Your Monitoring Stack

It's one thing to define what degradation looks like on a whiteboard, and quite another to get it running in production without your on-call team muting the alerts after two days. We've found the sweet spot is a layered approach that separates fast, noisy signals from slow, reliable ones. Don't try to build a single super-alert that catches everything. Instead, let your edge synthetics and RUM data handle the early whisper, and let your origin and PoP-level SLIs confirm it.

In practice, that means running lightweight synthetic checks every 30 seconds from at least 8-12 geographically diverse probes — not just your main markets. Each probe should fetch a small, cacheable object and a dynamic, uncacheable one. You're not just checking for 200 OK; you're tracking TTFB, time-to-last-byte, and TLS handshake time. When 2 or more probes in the same region see a 40% lift in p95 TTFB for 5 consecutive minutes, that's your early warning. It's not page-worthy yet, but it should fire a low-urgency Slack alert and auto-create a timeline event. We've cut our mean-time-to-detect partial outages from 18 minutes to under 4 minutes with this alone.

The mistake most teams make here is automating the remediation too early. If your auto-failover kicks in on that first whisper, you'll flap traffic and make things worse. This is exactly The CDN Automation Reliability Paradox: When Automation Causes More Outages in action — automation that triggers on weak signals creates the instability you're trying to prevent. Gate your automated actions behind a confirmation layer that requires agreement from at least two independent signal types, like synthetics plus RUM error rate, before it does anything disruptive.

# Example: Prometheus alert for regional CDN degradation (not just downtime)
# Requires metrics: cdn_request_duration_seconds, cdn_requests_total

- alert: CDNRegionalDegradationWarning
  expr: |
    (
      histogram_quantile(0.95, sum by (pop, region) (rate(cdn_request_duration_seconds_bucket[5m]))) 
      / 
      histogram_quantile(0.95, sum by (pop, region) (rate(cdn_request_duration_seconds_bucket[1h] offset 1h)))
    ) > 1.4
    and
    (
      sum by (region) (rate(cdn_requests_total{status=~"5.."}[5m])) 
      / 
      sum by (region) (rate(cdn_requests_total[5m]))
    ) > 0.008
  for: 5m
  labels:
    severity: warning
    signal: "degradation"
  annotations:
    summary: "Regional degradation detected in {{ $labels.region }} ({{ $labels.pop }})"
    description: "p95 latency is 40%+ above 1h baseline AND 5xx rate is >0.8% for 5m. Investigate before auto-failover."

Advanced Patterns: Correlating Signals and Troubleshooting False Positives

Once your basic degradation alerts are stable, the real power comes from correlation. A single PoP showing 1.2% 5xx errors might be a local cache node failing, but three PoPs in different regions all showing a 0.5% bump in 503s at the same time? That's almost certainly your origin or a global control plane issue. You need to build views that aggregate by PoP, by region, by cache status (HIT vs. MISS vs. REVALIDATE), and by provider if you're multi-CDN. We've seen cases where our primary CDN looked perfectly healthy on global aggregates (99.92% success), but the HIT ratio in Frankfurt had dropped from 94% to 71% — we were thundering our origin and didn't know it until origin latency spiked 300ms.

Troubleshooting false positives is where most degradation programs die. If you're getting woken up for a 60ms latency bump that customers don't notice, your thresholds are too tight. Don't just tweak the alert — look at the distribution. A lot of that noise comes from last-mile variability on mobile networks. Filter your RUM data to exclude the slowest 5% of connections or segment by connection type before you calculate percentiles. And always compare against a 7-day rolling baseline for the same hour, not just the last hour. Traffic patterns at 9am Monday look nothing like 9am Sunday, and you'll get false degradation signals every week if you ignore seasonality.

When a real degradation does fire, your response shouldn't be ad-hoc Slack debugging. The best teams have already codified the triage steps. If your detection is solid, the next bottleneck is execution, which is why you should connect these alerts directly to CDN Runbook Automation: Converting Manual Procedures into Executable Scripts. A degradation alert should automatically attach the relevant runbook — "Check origin concurrency, check cache purge queue, check provider status page" — and pull the last 15 minutes of PoP health into the incident channel. And don't forget to close the loop. Every degradation event, even the ones that didn't become outages, should feed into your CDN Postmortem Analysis: Learning from Incidents to Prevent Recurrence process so you can tune thresholds with real data, not guesses.

Frequently Asked Questions