Performance Percentile Analysis for CDN: Going Beyond Average Latency Numbers

Back in 2019, I was on call when a major e-commerce client called to complain that a third of their users were seeing checkout pages load in over 3 seconds. I pulled up our dashboard and it showed a global average latency of 48ms - green across the board. We argued for 20 minutes until they sent me their Real User Monitoring data. They were right, and our average was hiding a massive tail latency problem that was costing them conversions. That day completely changed how I think about performance metrics.

Why Averages Lie About Your CDN Performance

When you've operated CDNs for as long as I have, you learn that an average is often the most dangerous number on a dashboard. It gives you a single, clean figure that feels authoritative, but it collapses your entire latency distribution into one point that represents almost no one's actual experience. If 90% of your requests serve from cache in 20ms and 10% hit origin and take 800ms, your average will be around 98ms. You’ll tell yourself performance is fine, while one in ten users is having a terrible time. That's not an edge case, that's a segment of your audience you’re consistently failing.

The math behind it is simple, but the impact is subtle. Averages are incredibly sensitive to outliers in one direction and completely insensitive to the shape of your distribution. A bimodal distribution - where you have two distinct clusters of performance - looks identical to a normal distribution if you only look at the mean. I've seen this happen when a single PoP in Southeast Asia was saturated; global average moved by only 4ms, but users in that region saw p95 latency jump to over 900ms for hours before anyone noticed. If you're still reporting only averages to your stakeholders, you're flying blind to the experiences that actually drive churn and revenue loss.

That’s where percentile analysis comes in. Instead of asking "what's the typical latency?", we ask "what's the latency for the 95th or 99th percentile of users?" This shifts the conversation from a mathematical fiction to a guarantee. When you say p95 is 120ms, you’re saying 95% of your users experience 120ms or better. It’s a far more honest and actionable way to define performance metrics. And to get there reliably, you need more than a dashboard - you need proper access to raw events, which is why getting your CDN Real-Time Log Streaming Architecture: From Edge Events to Analytics in Seconds right is the foundation for anything beyond averages.

# Why mean hides the problem - quick Python check I run on new datasets
import numpy as np

# Simulated CDN latencies: 90% cache HIT (20ms), 10% origin MISS (800ms)
latencies = np.array([20]*900 + [800]*100)

print(f"Mean (Average): {np.mean(latencies):.1f}ms")
print(f"Median (p50): {np.percentile(latencies, 50):.1f}ms")
print(f"p95: {np.percentile(latencies, 95):.1f}ms")
print(f"p99: {np.percentile(latencies, 99):.1f}ms")

# Output:
# Mean (Average): 98.0ms - Looks OK
# Median (p50): 20.0ms - Looks great!
# p95: 800.0ms - Reality for 5% of users
# p99: 800.0ms - And it's not an outlier

Building a Percentile Pipeline From Raw Edge Logs

You can't calculate accurate percentiles from aggregated averages or pre-rolled summaries. I learned this the hard way when we tried to average percentiles from different PoPs - it’s mathematically invalid and gives you nonsense numbers. You need the raw latency for every request, or at least a sketch data structure that preserves the distribution. Modern CDN logging pipelines handle billions of events per day, so storing every single timing at full fidelity isn’t always an option, but you have to preserve distribution shape.

Most teams I work with now use one of two approaches. The first is streaming all edge timings into a columnar store like ClickHouse or Druid and using approximate quantile functions. The second, and my preference for cost-sensitive setups, is using sketches like t-digest or HDR Histogram at the edge. These let you compress the latency distribution into a tiny, mergeable structure that can calculate p50, p95, and p99 with less than 1% error while using kilobytes of memory instead of gigabytes. The trade-offs between full fidelity and sketching are exactly what you need to evaluate when designing your CDN Log Aggregation Architecture: Collecting Billions of Events Without Dropping Data, and if costs are biting, you’ll also want to review CDN Log Sampling Strategies: Reducing Storage Costs Without Losing Signal - but be warned, naive sampling destroys tail accuracy.

Don't sample randomly if you care about tails. If you sample 1% of logs uniformly, your p99 calculation is now based on 1% of your 1% - the error bars become huge. I’ve seen p99 estimates swing by 40% day-to-day just due to sampling variance. If you must sample, use head-based sampling weighted by latency or status code, or keep 100% of slow requests and sample the fast ones. For HTTP status and timing semantics, I've always found the HTTP Working Group Specifications useful for defining what to log consistently, and web.dev Articles provide excellent context on how these server-side percentiles map to user-centric metrics like LCP and INP.

-- Don't do this: averaging percentiles is wrong
-- SELECT avg(p95_latency) FROM pops -- INVALID

-- Do this in ClickHouse: calculate quantiles over raw data
SELECT
    quantilesExact(0.50, 0.90, 0.95, 0.99)(request_duration_ms) AS latency_quantiles,
    count() AS total_requests,
    countIf(request_duration_ms > 500) AS slow_requests
FROM cdn_edge_logs
WHERE timestamp >= now() - INTERVAL 1 HOUR
  AND cache_status IN ('HIT', 'MISS', 'BYPASS')
GROUP BY pop, cache_status
ORDER BY total_requests DESC;

-- For sketch-based approach (using t-digest)
-- SELECT tDigestQuantile(state, 0.95) FROM tdigest_states

How to Read Latency Distribution Beyond the Mean

Once you have p50, p95, and p99 time-series, you need to know how to interpret them without panicking over every spike. Here’s how I teach junior engineers to read them. p50, the median, is your typical user experience. It’s resistant to outliers and tells you how your cache and network perform under normal conditions. If p50 starts creeping up, you’ve got a systemic issue - maybe a bad cache config or congested upstream.

p95 is your canary for capacity and origin problems. In my experience, p95 is the most actionable percentile for a CDN. It filters out the absolute freak outliers but captures the consistent pain of users on slower networks, farther PoPs, or cache misses. When p95 diverges from p50, your distribution is widening - you’re serving two different classes of experience. p99 and p99.9 are your edge-case detectors. They’ll be noisy and spiky, but they catch TCP retransmits, TLS handshake failures, and origin timeouts that affect a small but vocal set of users. Don't alert on p99 alone, but use it for weekly SLO reviews.

The real insight comes from looking at the gaps between these percentiles, not just the absolute values. A healthy CDN usually has a tight distribution. When the gap between p50 and p95 is large, your performance is inconsistent. When p99 is an order of magnitude higher than p95, you have a long tail that needs investigation - often it’s origin fetches, uncacheable API calls, or a specific device/browser combination. Track these gaps over time, split by cache status, PoP, and content type, and you’ll find problems weeks before your average-based alerts would fire.

Putting Percentiles Into Production: From Query to Alerting

Knowing you need p95s is the easy part — wiring them into a pipeline that doesn't fall over at 2 AM is where most teams get stuck. If you're still pulling raw access logs into Elasticsearch and running percentile aggregations on the fly, you're burning money and you'll time out the moment you try to look at 30 days of data. We've seen that pattern kill dashboards for three different clients last year alone.

What actually scales is pre-aggregating with a sketch at the edge log processor. Don't store every request's TTFB if you don't have to. Use T-Digest or DDSketch to compress your distribution into a few hundred bytes per PoP, per 1-minute bucket. At ingestion time, your log shipper (Vector, Fluent Bit, whatever you're running) calculates the sketch and ships that. Your ClickHouse or Druid cluster then merges sketches at query time. You get p50, p90, p95, p99 in under 400ms across 10 billion requests, instead of waiting 45 seconds for a brute-force percentile_cont() to scan.

Here's the pattern we run in production for most CDN log pipelines. It works whether you're on Fastly, CloudFront, or Akamai writing to S3. The key is quantilesExactWeighted or quantileTDigest — don't use avg() anywhere in this pipeline. Pair this with segmented analysis from your Clickstream Analysis at CDN Edge: Understanding User Navigation Patterns to break percentiles down by asset type, not just PoP.

-- ClickHouse: Merge T-Digest sketches for fast percentile reads
-- Table: cdn_perf_1m (sketch column is TDigestState)

SELECT
  PoP,
  asset_type,
  quantilesTDigest(0.5, 0.90, 0.95, 0.99)(merge(s.latency_digest)) AS latency_ms,
  quantilesTDigest(0.95)(merge(s.ttfb_digest)) AS p95_ttfb_ms,
  countMerge(s.req_count) AS total_requests
FROM cdn_perf_1m AS s
WHERE timestamp >= now() - INTERVAL 7 DAY
  AND cache_status IN ('HIT', 'MISS')
GROUP BY PoP, asset_type
HAVING total_requests > 10000
ORDER BY p95_ttfb_ms DESC
LIMIT 20;

-- Alert rule: Fire only if p95 degrades >20% vs 7-day baseline
-- Prevents false positives from a 5ms jitter on a 42ms p50

Advanced Patterns: When Your Percentiles Are Lying to You

Once you've got the pipeline running, you'll notice something unsettling: your p50 can look perfect at 48ms while your p99 is sitting at 1.8 seconds. That's not a statistical quirk — that's your users hitting origin. The most common mistake we see is lumping HIT and MISS together. A 92% cache hit ratio sounds great until you realize the 8% of MISS traffic drives 70% of your p95+ latency. Always split your histograms by cache_status, colo, and content_type. Your API JSON endpoints will have a completely different distribution than your 2MB hero images.

Another gotcha is coordinating percentiles across PoPs. You can't just average the p95 from Ashburn and Singapore and call it your global p95. Percentiles aren't averageable. If Ashburn's p95 is 120ms and Singapore's is 340ms, your global p95 isn't 230ms — it's weighted by request volume, and the long tail in APAC will drag the true global number much higher. Merge the underlying sketches or raw histograms, then compute the percentile. If you're fusing performance data with behavioral data like User Journey Analytics from CDN Logs: Mapping the Path to Conversion, this weighting becomes critical — a slow p99 checkout page hurts revenue 100x more than a slow p99 favicon.

Finally, watch out for bimodal distributions. If your histogram shows two humps — one at 35ms and another at 850ms — you don't have a latency problem, you have two different populations. Usually it's HTTP/2 vs. HTTP/3, or IPv4 vs. IPv6, or bots vs. real users. Don't try to "fix" the average. Isolate the slower cohort and fix it at the source. We've traced bimodal humps to a single misconfigured origin shield in Frankfurt that added 600ms to 12% of MISS requests, which was invisible on the averaged dashboard but screamed at p95. For revenue-critical flows, tie this directly to Conversion Attribution from CDN Edge: First-Party Tracking Without Client-Side Scripts so you can prove that shaving 200ms off p95 actually moved conversion rate by 1.2%.

Frequently Asked Questions