CDN Performance Regression Detection: Automated Alerts for Degradation

Last winter we rolled out a small header normalization patch to a dozen edge PoPs and watched our dashboards stay green for two hours. Then support tickets started trickling in from Southeast Asia — video start times were up almost 800ms, but our global average TTFB hadn't moved. That incident cost us a full afternoon of rollback and post-mortem, and it taught me that if your CDN monitoring only looks at averages, you won't catch a performance regression until your users do.

Why Averages Hide Your Worst Regressions

I've spent the last decade staring at CDN time-series, and the fastest way to miss degradation is to average it away. A global mean blends traffic from well-connected US eyeballs with high-loss mobile networks in India or Brazil. You can have a 30% slowdown for 15% of your users and see barely a 2% bump in the global average. Percentiles tell a truer story. If you aren't tracking at least p50, p90, and p95 for TTFB and download completion time per PoP, per ASN, and per content type, you don't have degradation detection — you have wishful thinking.

The other trap is sampling too coarsely. Five-minute aggregates are fine for capacity planning, but for regression detection they're too slow and too smooth. A bad config push or a congested upstream can spike tail latency within seconds. We learned to separate our metrics into two layers: high-cardinality, high-frequency signals for alerting (10-30 second buckets, percentile-based) and downsampled aggregates for dashboards. That separation also keeps your alerting pipeline from drowning in cardinality. And don't forget that cache state matters — a drop in hit ratio can look like a latency regression when it's really an origin problem. Correlating edge latency with cache status and origin fetch time avoids chasing the wrong root cause. For a deeper look at transport-level effects, I've often pointed teams to RTT Measurement Methodology for CDN: Getting Accurate Round-Trip Times because noisy RTT data will poison any regression model you build.

# Simple rolling regression check using median and MAD
# Flags when p95 TTFB deviates beyond baseline + 3*MAD
import numpy as np

def is_regression(baseline_p95, current_window):
    median = np.median(baseline_p95)
    mad = np.median(np.abs(baseline_p95 - median)) * 1.4826
    # Require both statistical and practical significance
    threshold = median + max(3 * mad, 0.15 * median)
    current_p95 = np.percentile(current_window, 95)
    return current_p95 > threshold, current_p95, threshold

# Example: 7-day baseline vs last 5 minutes
baseline = np.array([112, 118, 115, 121, 119, 117, 120, 114, 116]) # ms
current = np.array([118, 135, 142, 138, 155, 149, 132]) # ms recent samples
flagged, p95, thresh = is_regression(baseline, current)
print(f"p95={p95:.1f}ms threshold={thresh:.1f}ms flagged={flagged}")

Building a Baseline That Doesn't Cry Wolf

Every team I've worked with wants automated alerts, and every team gets burned by noisy alerts in the first month. The culprit is usually a naive baseline — like comparing this Tuesday 2pm to last Tuesday 2pm, or using a fixed 200ms threshold for all PoPs. CDN traffic has seasonality, weekday/weekend patterns, and PoP-specific quirks. A PoP in Johannesburg doesn't behave like one in Frankfurt, and you can't alert them the same way. What works better is a rolling, PoP-local baseline that updates continuously but resists learning from the regression itself.

We settled on a 7-day rolling median with a median absolute deviation (MAD) band, computed separately for each PoP and content class (static, small API, large video segment). MAD is far more robust than standard deviation when you have bursty outliers, which CDNs always do. We also add a minimum practical delta — say 15% or 20ms, whichever is larger — so tiny statistical wiggles don't page anyone at 3am. To prevent the baseline from absorbing a slow-burn regression, we freeze baseline updates when an alert is active and we use a double-window approach: a long window (7 days) for the baseline and a short window (3-5 minutes) for evaluation. The HTTP Working Group Specifications and guidance on timing metrics in web.dev Articles are useful references when you're deciding which timestamps to actually compare — responseStart versus responseEnd changes your baseline completely.

One mistake I made early on was alerting on synthetic checks alone. Synthetics are clean and consistent, but they don't see real user network variance. Real User Monitoring (RUM) gives you the distribution, synthetics give you a control. We now require both to agree before paging: if RUM p95 degrades and our synthetic probe from the same region confirms it, it's real. If only RUM moves, we warn but don't wake anyone up. That two-signal rule cut our false positives by about 60% without missing a real incident last year.

# Prometheus alerting rule for p95 TTFB regression per PoP
groups:
  - name: cdn_regression
    interval: 30s
    rules:
      - alert: CDNEdgeP95Regression
        expr: |
          (
            histogram_quantile(0.95, sum by (pop, le) (rate(edge_request_duration_seconds_bucket[3m])))
            >
            (
              quantile_over_time(0.5, edge_p95_ttfb_seconds[7d] offset 5m)
              + max(
                3 * stddev_over_time(edge_p95_ttfb_seconds[7d] offset 5m),
                0.15 * quantile_over_time(0.5, edge_p95_ttfb_seconds[7d] offset 5m)
              )
            )
          ) and on (pop) (
            histogram_quantile(0.95, sum by (pop, le) (rate(synthetic_probe_duration_seconds_bucket[3m]))) > 1.15 * quantile_over_time(0.5, synthetic_probe_duration_seconds[7d])
          )
        for: 5m
        labels:
          severity: page
        annotations:
          summary: "p95 TTFB regression detected in {{ $labels.pop }}"
          description: "Current p95 {{ $value }}s exceeds adaptive baseline for 5m. Check recent config or upstream."

Choosing Signals That Predict User Pain, Not Just Server Stats

Not all metrics are equal for automated alerts. CPU, connections, and even cache hit ratio are lagging or indirect. Users feel time-to-first-byte, time-to-last-byte, and rebuffer rate. When we rebuilt our alerting at my previous company, we ranked signals by how well they correlated with abandonment and playback failure, not by how easy they were to collect. Edge TTFB p95 per PoP was number one, followed by origin fetch p95 and segment download throughput for video. Transport metrics like retransmits and congestion window drops are fantastic for diagnosis, but they're too noisy to page on directly — use them as supporting evidence after the main alert fires.

You also need to account for how TCP behavior shapes what you measure. A change in initial congestion window or window scaling can shift throughput without touching TTFB, so your regression might show up only on large objects. That's why we split alerts by object size bucket and keep an eye on throughput for >1MB responses. If you're tuning those transport parameters, TCP Window Scaling and Its Impact on CDN Throughput: Why Defaults Fall Short and Bandwidth-Delay Product and CDN Tuning: Filling the Pipe Without Overwhelming It are solid primers — getting the BDP math wrong will make every downstream alert look like a regression when it's just under-utilization. My rule of thumb: alert on user-centric latency first, confirm with throughput and error rate, then use TCP stats to pinpoint the fix.

Putting Detection Into Production: From Query to PagerDuty

Here's where most teams stall — they've got the right percentiles and a solid baseline in a notebook, but nothing's actually watching production at 2 AM. You don't need a fancy ML platform to start. We've seen the best results with a dead-simple pipeline: every 5 minutes, query your edge logs or RUM, compute p50/p95/p99 by PoP and content type, compare it against your rolling baseline, and emit a normalized anomaly score. If that score breaches for two consecutive windows, you page. That two-window rule alone cut our false positives by 62% overnight.

The trick is to alert on the relative change, not the absolute value. A 20ms jump on a 40ms p50 in Ashburn is a 50% regression — users feel it. That same 20ms on a 280ms p99 in Jakarta is noise. We use a modified z-score against a 7-day rolling median with a MAD (Median Absolute Deviation) denominator, per PoP, per percentile. It auto-adjusts for that slow PoP that’s always 30ms hotter and it won’t freak out during a Super Bowl traffic spike where everything shifts together. If you're still alerting on static thresholds like "p95 > 150ms", you're going to burn out your on-call before you catch a real regression. For a refresher on why you need all three percentiles in that calculation, see our breakdown on CDN Percentile Latency Analysis: What p50, p95, and p99 Actually Tell You.

# Example: Prometheus-style alert for p95 regression per PoP
# Fires only if p95 is >25% above 7d median for 10m
- alert: CDN_P95_Regression
  expr: |
    (
      histogram_quantile(0.95, sum by (pop, le) (rate(edge_request_duration_seconds_bucket[5m])))
      /
      quantile_over_time(0.5, histogram_quantile(0.95, sum by (pop) (edge_request_duration_seconds_bucket[7d]))[7d:5m])
    ) > 1.25
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "p95 regression detected in {{ $labels.pop }}: {{ $value | humanizePercentage }} of baseline"

Advanced Patterns: Taming Seasonality, Deploys, and Ghost Regressions

Once your basic detector is stable, you'll hit the weird stuff. The first is deploy-correlated jitter. We had a case where every config push added 8-12ms of CDN Jitter Measurement and Its Impact on User Experience for about 90 seconds while connections re-warmed — not a regression, just TCP and TLS session cache churn. If you don't tag your alerts with deploy IDs, you'll chase ghosts every Tuesday. Now we automatically annotate our Grafana with Fastly / CloudFront config versions and suppress regression alerts for 3 minutes post-deploy unless p99 degrades by more than 40%. It's saved us at least four war-room escalations this quarter alone.

The second ghost is subtler: packet reordering that looks like latency. Your p95 TTFB spikes 15-25ms, your origin looks clean, but throughput for large assets craters. That’s classic reordering on a congested transit link, not your cache. Traditional latency alerts will blame your edge stack when the real culprit is a single peer in Frankfurt dropping packets out of order. We now pair every latency regression alert with a companion check on retransmits and out-of-order packet counters from our eBPF sidecars. If you see that pattern, don’t roll back your code — read up on Packet Reordering Effects on CDN Performance: The Hidden Throughput Killer and fail that path over instead.

Finally, don't forget to detect slow regressions. A 1% per-day creep in p95 won't trip a 25% threshold, but over three weeks you're 20% slower and conversion's down 2%. We run a second, slower detector nightly: linear regression over the last 14 days of p95 per PoP. If the slope is positive with p < 0.05 and total drift > 10%, it opens a low-severity ticket. That's how we caught a gradual SSD wear-out in our Chicago cache that was adding 0.8ms/day to cache-miss latency. No one felt it day-to-day, but our automated ticket did.

Frequently Asked Questions