CDN Performance Anomaly Detection with Machine Learning: Spotting Issues Before Users Notice

At 2:47 AM last summer, our edge PoP in Singapore started serving 12% slower TTFBs and nobody paged. The dashboards were green - CPU, cache hit ratio, origin errors all within thresholds. But users on 4G connections were already abandoning carts, and by the time our static 500ms alert fired 40 minutes later, we'd lost a measurable chunk of revenue. That's the gap that pushed us to stop trusting fixed thresholds entirely.

Why Static Thresholds Can't Catch Slow-Burn CDN Degradation

I've spent a decade watching CDN monitoring evolve from simple Nagios checks to Grafana dashboards with a hundred graphs, and the failure mode hasn't really changed. We set a threshold - say p95 latency < 300ms - and assume anything under it is healthy. The problem is that CDN performance isn't stationary. Latency at 3 AM in Frankfurt looks nothing like latency at 8 PM during a Champions League stream. If you set thresholds loose enough to avoid night-time false positives, you'll completely miss a 30% degradation during peak. If you tighten them, your on-call team burns out in a week.

Anomalies in CDN contexts are almost always multivariate and subtle. It's rarely a single metric spiking. It's a pattern: cache hit ratio dipping 2%, origin fetch time climbing 15ms, and RTT Measurement Methodology for CDN: Getting Accurate Round-Trip Times showing a 5ms increase on a specific ASN, all at once. Traditional monitoring treats each metric in isolation, so none of them individually crosses an alert line. ML monitoring works because it learns the joint distribution. An autoencoder that has seen six months of normal PoP behavior knows that those three small shifts happening together have never occurred without an impending incident.

We learned this the hard way when our Bandwidth-Delay Product and CDN Tuning: Filling the Pipe Without Overwhelming It tuning looked perfect in synthetic tests but caused micro-bursts on real user connections. Throughput stayed high, so throughput alerts didn't fire, but retransmits crept up. Only a model looking at throughput, cwnd, and retransmit rate together flagged it pre-impact.

# Training a per-PoP baseline that captures seasonality and ASN variance
import pandas as pd
from sklearn.ensemble import IsolationForest

# Features engineered from 1-minute edge aggregates
features = ['p50_ttfb', 'p95_ttfb', 'cache_hit_ratio', 
            'origin_fetch_p95', 'tcp_retrans_pct', 'ewma_rtt_delta']

# Fit per PoP + per hour-of-week to handle diurnal/weekly seasonality
# contamination is tuned from historical incident rate (~0.8%)
model = IsolationForest(
    n_estimators=200,
    contamination=0.008,
    max_samples=0.7,
    random_state=42
)

df['hour_of_week'] = df.timestamp.dt.dayofweek * 24 + df.timestamp.dt.hour
for (pop, how), group in df.groupby(['pop_id', 'hour_of_week']):
    model.fit(group[features])
    scores = model.decision_function(group[features])
    # Persist model for real-time scoring in Flink
    save_model(pop, how, model)

Feature Engineering That Mirrors What Users Actually Feel

Most teams start their ML monitoring project by throwing every metric they collect into a model and hoping it learns something. We've tried that. It doesn't work, and it creates a model you can't debug at 3 AM. The features have to be grounded in how browsers and the transport layer actually behave. Raw CPU or bytes-served are poor signals - they're lagging indicators. What matters is what web.dev Articles and the HTTP Working Group Specifications emphasize: time to first byte, content download time, connection reuse, and protocol efficiency.

For our proactive CDN pipeline, we settled on four feature families. First, client-perceived performance: p50/p95/p99 TTFB and TTFB variance per ASN and per content-type. Second, transport health: smoothed RTT, RTT variance, retransmit percentage, and congestion window distribution. That second family is why understanding TCP Window Scaling and Its Impact on CDN Throughput: Why Defaults Fall Short is critical - if you don't normalize throughput anomalies by window size, you'll flag every large-object download as anomalous. Third, cache and origin behavior: hit ratio, stale-while-revalidate ratio, origin p95, and 5xx rate. Fourth, request shape: HTTP/2 vs HTTP/3 mix, TLS version distribution, and cache-key cardinality. The magic isn't in having 200 features; it's in having 15 that are de-noised.

De-noising is where most projects fail. CDN metrics are bursty and heavy-tailed. A single 10-second origin slowdown shouldn't retrain your baseline. We apply a two-stage pipeline: a 5-minute exponentially weighted moving average to smooth micro-bursts, then seasonal-trend decomposition (STL) to subtract expected diurnal patterns before scoring. Without that, your model will just learn that "traffic goes up at 9 AM" and flag everything else as noise. For performance prediction, we don't predict absolute latency - we predict the residual after removing seasonality, which makes anomalies pop with 10x more signal.

-- Real-time feature pipeline in Flink SQL (1-min tumbling windows)
SELECT
  pop_id,
  asn,
  TUMBLE_START(event_time, INTERVAL '1' MINUTE) AS window_start,
  percentile(ttfb, 0.95) AS p95_ttfb,
  avg(retransmits * 1.0 / packets_sent) AS tcp_retrans_pct,
  -- EWMA to suppress micro-bursts, alpha=0.3 ~ 5-min half-life
  ewma(avg(smoothed_rtt), 0.3) OVER (
    PARTITION BY pop_id, asn ORDER BY window_start
  ) AS ewma_rtt,
  percentile(origin_fetch_ms, 0.95) AS origin_p95,
  count(*) FILTER (WHERE cache_status = 'HIT') * 1.0 / count(*) AS hit_ratio
FROM edge_access_log
GROUP BY pop_id, asn, TUMBLE(event_time, INTERVAL '1' MINUTE)
HAVING count(*) > 500; -- floor to avoid low-sample variance

Picking Models for Operational Reality, Not Leaderboard Scores

When we benchmarked models offline, a deep LSTM autoencoder beat Isolation Forest on pure F1 score by about 6%. In production, we rolled back the LSTM within two weeks. It needed 40ms per inference on GPU, required 30 days of continuous training data to converge per PoP, and when it fired, we couldn't explain why to the network team. For anomaly detection that has to run on 150+ PoPs with scoring latency under 100ms, operational cost matters more than a few points of accuracy. We now use a tiered approach and choose the simplest model that meets our detection SLO.

Isolation Forest remains our workhorse for per-PoP, low-dimensional scoring - it trains in seconds, scores in microseconds on CPU, and handles non-linear interactions without feature scaling headaches. For cross-PoP global anomalies like a bad config rollout, we use a lightweight variational autoencoder on aggregated features, but we distill it to ONNX and run it on CPU. And for performance prediction - not just detection - we use a gradient-boosted quantile regressor that forecasts p95 TTFB for the next 10 minutes. If the predicted p95 is 20% above seasonal expectation with >0.85 confidence, we page before users see it. That's shifted us from detection in ~4 minutes to prediction with a 7-11 minute lead time.

We evaluate everything on three criteria I'd recommend you adopt: time-to-detect on injected faults, false positive rate per PoP per week, and alert explainability. A model that catches an anomaly 30 seconds faster but doubles false positives will get silenced by engineers, which is worse than no model at all. Below is how our current production candidates actually compare on real edge traffic from the last quarter. These aren't lab numbers - they're measured on shadow traffic with injected latency and packet loss faults.

From Prototype to Production: Wiring ML Detection Into Your CDN Telemetry Pipeline

We've all seen it — a model that scores 0.98 AUC in a Jupyter notebook and then completely falls apart when you point it at live CDN logs. The gap isn't the algorithm, it's the plumbing. In production, you aren't scoring a clean CSV every hour; you're chewing through millions of edge requests per minute, and your detection loop has to keep up without adding latency to the delivery path itself.

What actually works is a sidecar scoring pipeline that taps your existing telemetry stream — whether that's Kafka, Kinesis, or Vector — and aggregates in 60-second tumbling windows per PoP and per cache status (HIT, MISS, PASS). Don't try to score every request. You'll drown. Instead, emit windowed features: rolling p50/p95/p99 for TTFB and download time, 1-minute CDN Jitter Measurement and Its Impact on User Experience (stddev of inter-arrival delta), origin fetch error rate, and cache hit ratio delta vs. the same minute last week. That last one is crucial — comparing to last week at the same time kills 90% of your false positives from diurnal traffic patterns. In our deployment, this cut alert noise from ~45 alerts/day with static thresholds to 3-4 actionable anomalies per week, with a median detection lead time of 4-7 minutes before our synthetic monitors tripped.

Keep your model artifact small and your inference fast. We run an Isolation Forest (contamination=0.015) for baseline and a lightweight LSTM-Autoencoder for PoPs with enough history (>30 days). Both score in <40ms on a 2-vCPU sidecar. Your pipeline should also version features and log every scored window to S3/ClickHouse so you can replay and retrain without guessing. Here's the stripped-down version of our streaming scorer:

# streaming_anomaly_scorer.py - scores 60s windowed CDN telemetry
import numpy as np
from collections import deque

WINDOW = 60  # seconds
HISTORY = deque(maxlen=10080)  # 7 days of 1-min windows

def build_features(window_df, history_df):
    # Latency percentiles tell you WHERE users feel it
    p50, p95, p99 = np.percentile(window_df['ttfb_ms'], [50, 95, 99])
    # Jitter + p99/p50 ratio catches slow-burn degradation static thresholds miss
    jitter = window_df['ttfb_ms'].std()
    tail_ratio = p99 / max(p50, 1)
    
    # WoW delta features - critical for not alerting on normal peaks
    hist_median_p95 = history_df[history_df['minute_of_week'] == window_df['minute_of_week'].iloc[0]]['p95'].median()
    p95_wow_delta = (p95 - hist_median_p95) / max(hist_median_p95, 1)
    
    origin_error_rate = (window_df['cache_status'] == 'MISS_ERROR').mean()
    hit_ratio = (window_df['cache_status'] == 'HIT').mean()
    
    return np.array([p50, p95, p99, jitter, tail_ratio, p95_wow_delta, origin_error_rate, hit_ratio])

def score_window(features, iso_forest_model, threshold=-0.35):
    # Isolation Forest decision_function: lower = more anomalous
    score = iso_forest_model.decision_function([features])[0]
    is_anomaly = score < threshold
    # Require p95 WoW delta > 15% to suppress noisy low-traffic PoPs
    if features[5] < 0.15 and score > -0.5:
        is_anomaly = False
    return score, is_anomaly

Advanced Patterns and Troubleshooting: When “Healthy” Dashboards Lie

Here’s where senior teams get burned: your averages look fine while a subset of users gets hammered. If you’re only watching regional p50, you’ll miss the classic slow-burn — p50 flat at 42ms, p95 creeping from 110ms to 165ms (+50%) over 35 minutes, and p99 spiking past 480ms. That's not a network blip; it's usually origin congestion or a bad cache key causing selective MISS storms. This is why CDN Percentile Latency Analysis: What p50, p95, and p99 Actually Tell You matters more than any single “average latency” graph. We flag any window where the p99/p50 ratio exceeds 4.5x and WoW p95 delta >18% for more than 3 consecutive windows — that pattern alone catches ~70% of our pre-outage degradations before error rates move at all.

Another pattern that’ll drive you crazy if you don’t instrument for it is jitter coupled with packet reordering. Standard throughput and loss metrics won’t show it, but users on mobile and last-mile congested networks will see stuttering video and stalled downloads. If your anomaly detector ingests only HTTP logs, you’re blind to it. We now join edge TCP stats (retransmits, out-of-order packet count, RTT variance) into the feature window every 60s. When jitter jumps >25ms and out-of-order rate climbs above 1.8%, even with stable p50, you’re often looking at an upstream peering flap or a bad congestion control tuning rollout — see Packet Reordering Effects on CDN Performance: The Hidden Throughput Killer for a deep dive. Our model learned to weight this combo heavily after we fed it two incidents where reordering-induced rebuffering preceded a full PoP failover by 12 minutes.

Troubleshooting false positives? Don't just tune the threshold — tune the context. The top three fixes we’ve made: 1) Suppress anomalies during known deployment windows by joining your CI/CD event feed — 40% of our early false positives were mid-deploy cache purges. 2) Normalize by traffic volume; a PoP serving 200 req/min will have wildly noisy percentiles vs. one serving 80k req/min. We disable p99-based scoring below 1,500 req/min/window. 3) Always log SHAP or feature contributions with each alert. If your on-call can’t see *why* the model fired (“p95 WoW +22%, jitter +31ms, hit ratio -8%”), they won’t trust it and they’ll mute it. Trust is the real production metric.

Frequently Asked Questions