CDN Network Quality Scoring: Building a Composite Performance Metric

Back in 2019 I was staring at a Grafana dashboard that was completely green. P50 latency was down, cache hit ratio was up, and our origin error rate was near zero. Then our largest video customer called to say their viewers in Southeast Asia were seeing constant rebuffers. That's the moment I stopped trusting single-metric health checks. A CDN can look perfect on one chart and be failing users on another, which is exactly why we needed a proper network quality score that tells the truth.

Why Single Metrics Can't Capture CDN Quality

For years I've watched teams chase the wrong thing because they picked one number to represent CDN quality. One team optimizes for cache hit ratio and ends up serving stale content longer than they should. Another obsesses over edge latency and forgets that a 20ms TTFB doesn't help if throughput collapses after the first 50KB. I’ve been guilty of this myself — we once tuned an entire PoP for lower RTT, only to find we’d increased retransmits by 18% because our congestion control was too aggressive.

The problem is that user experience isn't a single dimension. A composite metric forces you to look at latency, throughput, stability, and availability together. If you only track latency, you miss the user on a high-latency but high-bandwidth fiber link who can't fill the pipe. If you only track bandwidth, you miss the user on a lossy mobile connection where every packet drop kills performance. That's why RTT Measurement Methodology for CDN: Getting Accurate Round-Trip Times matters so much — if your RTT data is noisy or based on TCP handshake only, your entire performance scoring model starts with bad inputs. You can't build a reliable score on shaky measurements, and as the HTTP Working Group Specifications show, how you measure timing (TTFB vs. full response) changes the story completely.

# Naive composite score - the first version we shipped (and rewrote)
def calculate_quality_score(metrics):
    # All inputs normalized to 0-100 where 100 is perfect
    # Don't do this in production - no clipping or outlier handling
    weights = {
        'ttfb_score': 0.30,
        'throughput_score': 0.30,
        'loss_score': 0.20,
        'availability_score': 0.20
    }
    score = sum(metrics[k] * weights[k] for k in weights)
    return round(score, 2)

# Example: Good latency but poor loss handling = mediocre score
sample = {'ttfb_score': 92, 'throughput_score': 85, 'loss_score': 45, 'availability_score': 99}
print(calculate_quality_score(sample))  # 82.1 - not 95+ as latency alone would suggest

Selecting and Normalizing Signals for Your Composite Metric

Building a useful composite metric starts with picking signals that actually reflect what users feel, not just what's easy to collect. In my experience, four families cover 90% of cases: responsiveness (TTFB, RTT), throughput (goodput, not just link speed), stability (packet loss, retransmit rate, jitter), and reliability (availability, 5xx rate, cache hit quality). You don't need twenty signals. We started with twelve and ended up with five because the extra ones were just correlated noise that made the score harder to explain to product teams. Start small, validate against real user telemetry like Core Web Vitals or video rebuffer rate, then add complexity only if correlation improves.

Normalization is where most teams mess up. You can't average a 40ms TTFB with a 25 Mbps throughput and a 0.8% loss rate — the units don't mix. You have to convert each signal to a 0-100 quality scale first. I've found percentile-based clipping works better than pure min-max, because one bad PoP with 2000ms TTFB will skew your entire scale if you aren't careful. We also use an inverted scale for negative metrics: lower loss must map to a higher score. For throughput, you have to account for the Bandwidth-Delay Product and CDN Tuning: Filling the Pipe Without Overwhelming It and whether your TCP Window Scaling and Its Impact on CDN Throughput: Why Defaults Fall Short is configured correctly — otherwise you'll penalize a path for low throughput that’s actually a window size limitation, not network quality. The web.dev Articles on LCP and TTFB provide solid guidance on what thresholds users actually perceive as good or poor.

import numpy as np

def normalize_signal(value, p5, p95, invert=False, clip=True):
    """Normalize raw metric to 0-100 using percentile bounds."""
    # p5/p95 derived from 30-day rolling window per PoP/region
    if clip:
        value = np.clip(value, p5, p95)
    # Linear scale to 0-100
    normalized = (value - p5) / (p95 - p5) * 100
    if invert:
        # Use for loss, latency, error rate where lower is better
        normalized = 100 - normalized
    return round(normalized, 1)

# Example: TTFB 85ms where p5=20ms, p95=250ms -> higher is worse, so invert
print(normalize_signal(85, 20, 250, invert=True))  # 71.7
# Throughput 45 Mbps where p5=5, p95=100 -> higher is better
print(normalize_signal(45, 5, 100, invert=False))  # 42.1

How We Weight Signals to Match User Experience

Weights are opinions encoded as math, so make them explicit and testable. We didn't guess our weights; we derived them by correlating each normalized signal against real outcomes — video rebuffer ratio, page abandonment, and origin retry rate — using a simple regression over 60 days of per-ASN data. For our video-heavy workload, throughput and loss mattered far more than raw TTFB, while for our API customers it was the opposite. Your network quality score should have different weight profiles per product, or you'll end up optimizing for an average user who doesn't exist. I recommend starting with an equal-weight model, then shifting weights based on which signal best predicts user complaints in your post-incident reviews.

You also need to decide how sensitive the score is to outliers. We apply a non-linear penalty for tail behavior: if packet loss exceeds 2%, its component score drops exponentially rather than linearly, because users don't perceive 0.5% vs 1% loss the same way they perceive 1% vs 3%. We also update weights quarterly, not daily, to avoid chasing noise. One mistake I made early on was letting the score auto-adjust weights every hour based on variance — it created a feedback loop where a flapping signal got down-weighted automatically, hiding the very problem we needed to see. Keep humans in the loop for weight changes and document why a signal is weighted at 0.35 and not 0.25.

Putting It Together: Implementing the Scoring Pipeline in Production

You've got your signals and your weights — now you've got to make it run reliably every 30-60 seconds across thousands of edge nodes without melting your telemetry pipeline. We ingest raw signals at the PoP level via Vector into Kafka, then aggregate in ClickHouse on a 1-minute tumbling window. Don't try to score per-request; you'll drown in noise and storage costs. Instead, we compute windowed aggregates (median TTFB, p95 latency, loss rate, etc.) per PoP / per ASN, then normalize and score. That keeps cardinality manageable and still gives you enough granularity to catch a single congested peer or a bad cache node.

Normalization is where most teams trip up. We don't use raw values. Each signal gets clipped and scaled to a 0-100 quality sub-score where 100 is perfect. For latency, for example, we clip TTFB at 50ms (100 points) and 500ms (0 points) and apply a logarithmic decay in between — because users don't feel the difference between 50ms and 70ms, but they absolutely feel 250ms vs. 400ms. For loss, 0% is 100, 0.1% is 85, and anything over 2% is 0. Jitter gets the same treatment, which is why understanding CDN Jitter Measurement and Its Impact on User Experience is critical before you set those thresholds. And don't just use p50 for latency — if you're only scoring the median, you'll miss the tail pain that drives rebuffers. Our latency sub-score blends p50 (30%), p95 (50%), and p99 (20%) as we detailed in our CDN Percentile Latency Analysis: What p50, p95, and p99 Actually Tell You.

Once normalized, the weighted composite is dead simple. We run it as a materialized view so every PoP gets a fresh score each minute, and we alert on a 5-minute rolling average dropping below 85 for more than 3 minutes. That dampens flapping but still catches real incidents 4-5x faster than alerting on any single metric. Here's the core logic we run in production:

def calculate_cdn_quality_score(metrics):
    # metrics: dict with normalized 0-100 sub-scores
    weights = {
        'latency': 0.40,      # p50/p95/p99 blended
        'loss': 0.20,
        'jitter': 0.15,
        'throughput': 0.15,
        'availability': 0.10
    }
    
    # Clip and invert where needed - lower raw = higher score already handled in normalization
    sub_scores = {
        k: max(0, min(100, metrics[k])) for k in weights
    }
    
    # Weighted composite 0-100
    composite = sum(sub_scores[k] * weights[k] for k in weights)
    
    # Penalty for tail degradation: if p99 sub-score < 40, cap composite at 75
    if metrics['p99_latency_score'] < 40:
        composite = min(composite, 75)
    
    return round(composite, 1)

When Your Score Looks Good But Users Are Still Hurting

If there's one thing I've learned running this at scale, it's that a composite score will lie to you if you let it. The most common failure mode is a healthy-looking 92 while real users are buffering. Nine times out of ten, it's because your weights are hiding tail latency or you're averaging away regional pain. We once had a PoP in Southeast Asia scoring 88/100 globally, but p95 for Telkomsel users was 1.8s due to a congested transit link. The fix wasn't tuning the formula — it was adding ASN-level scoring and not rolling up to a global average until after alerting. Always score at the most granular level you can, then aggregate up.

Another sneaky culprit is packet reordering. It won't show up as loss, and latency might look fine, but throughput collapses and your score stays green. Traditional loss + latency weights completely miss it. We now include a reordering ratio (out-of-order packets / total packets) as a 5% penalty factor inside the throughput sub-score. If reordering exceeds 0.5%, we dock up to 20 points from throughput. It's a small weight that prevents a big blind spot — you can read more on why this matters in Packet Reordering Effects on CDN Performance: The Hidden Throughput Killer. Without it, we saw cases where single-stream throughput dropped 30-40% while our old score didn't budge.

Finally, don't set it and forget it. User expectations and traffic mixes drift. We recalibrate weights quarterly by regressing our composite score against real user QoE — specifically, video rebuffer ratio and page LCP. If the correlation (R²) drops below 0.75, we know our weights are stale. We also run a simple anomaly detector on the score itself: a Z-score > 2.5 against a 7-day rolling baseline per PoP triggers a "score behavior changed" alert, even if the absolute score is still above 85. That catches slow-burn degradations, like a cache fill ratio slipping from 94% to 89% over two weeks, that you'd otherwise miss until it becomes an outage.

Frequently Asked Questions