CDN Latency Measurement Methodology: Getting Accurate Numbers That Matter

Last year a customer showed me their Grafana dashboard screaming about 180ms edge latency while our own monitoring insisted everything was a crisp 22ms. Same PoP, same hour, completely different stories. We weren't lying and neither were they - we were just measuring two entirely different things and calling them both "latency." If you've run a CDN for any length of time, you've had this argument, and it usually ends with someone realizing their numbers never mattered in the first place.

Why ICMP and Simple Pings Lie About Real CDN Performance

For years my team's default for RTT measurement was simple: ping the edge, graph it, alert if it spikes. It's clean, it's cheap, and it's almost useless for understanding CDN performance. ICMP echo requests are handled on a completely different path than your customer's actual traffic. On most modern routers ICMP is rate-limited or punted to a slow-path CPU, while TCP/UDP traffic is forwarded in hardware. I've seen cases where ICMP RTT was 45ms and the TCP handshake to port 443 on the same anycast IP was 12ms, simply because the ICMP reply was deprioritized three hops upstream.

There's a second, sneakier problem: path divergence. In an anycast CDN, your probe and your user's request might not even hit the same machine. When we use ECMP Routing in CDN Networks: Equal-Cost Multi-Path Traffic Distribution at our aggregation layer, the hash for ICMP (IP only) and the hash for TCP (5-tuple) can land on different spine links with different congestion. Add MPLS in CDN Backbone Networks: Label Switching for Predictable Performance in the backbone and the label-switched path for your probe might be explicitly engineered away from the path carrying real cache-fill traffic. You're not measuring network latency anymore, you're measuring a ghost path that no user ever traverses.

That's why we stopped treating ping as ground truth and shifted to measuring the handshake the user actually pays for. For us, the only defensible definition of network latency is the time taken to complete a TCP handshake or, even better, the kernel's own RTT estimate for an established connection. It's not as pretty as a ping graph, but it doesn't lie. The Linux kernel already tracks this for you if you ask it nicely, and it reflects retransmits, congestion, and path asymmetry the way a single ICMP packet never will.

# Get real TCP RTT from the kernel - not ICMP - using TCP_INFO
# Requires a real TCP connection, works great from your synthetic probes
import socket
import struct

# TCP_INFO struct field 0 is tcpi_rtt (microseconds) on recent kernels
TCP_INFO = 11
fmt = 'B' * 7 + 'I' * 24  # simplified; use proper struct for your kernel
# In practice, use: ss -ti or getsockopt parsing library

sock = socket.create_connection(("cdn.example.net", 443))
info = sock.getsockopt(socket.IPPROTO_TCP, TCP_INFO, 1024)
# Parse tcpi_rtt and tcpi_rttvar - gives you smoothed RTT in us
# Compare this to `ping` and watch the gap appear on congested paths

# Quick shell equivalent for debugging:
# ss -ti '( dport = :443 )' | grep -E 'rtt:|rto:'

Building a Probe Mesh That Measures What Users Feel

Once you know what to measure, you have to decide where to measure from. I learned this the hard way back in 2018 when we deployed 40 new probes all in AWS us-east-1 and declared our p95 latency "solved." Our users, mostly on residential broadband in Brazil and India, didn't get the memo. Cloud vantage points have fantastic peering that real eyeball networks don't. If all your latency measurement comes from well-connected data centers, you're systematically underestimating last-mile congestion, carrier NAT, and the bufferbloat on that cheap home router.

A realistic synthetic mesh needs three kinds of vantage points: backbone PoPs doing edge-to-edge checks, cloud probes for baseline stability, and, crucially, probes inside eyeball networks - RIPE Atlas, residential Raspberry Pis, or commercial RUM data. And you need to separate your measurements by connection state. A cold TCP+TLS 1.3 handshake to a PoP that has to go back to origin for a cache miss is a completely different number than a warm HTTP/2 connection reusing a session to a hot cache. We tag every single probe with `cache_status:HIT/MISS`, `reused_connection:true/false`, and `protocol:h2/h3` so we don't average apples and oranges into a meaningless mush.

I've also found you have to control for Segment Routing for CDN Traffic Engineering: Source-Routed Path Selection policies when you schedule probes. If your traffic engineering is actively shifting user prefixes between paths based on latency, a naive probe that always hits the same prefix will miss the shift. We now probe a rotating set of prefixes that match our actual traffic distribution, not just the PoP's anycast address. The HTTP Working Group Specifications are surprisingly helpful here for defining what TTFB and connect timing should actually include, so your `curl` timings map to what a browser reports.

# curl timing that separates network from server - don't just log total_time
curl -w @- -o /dev/null -s https://cdn.example.net/asset.jpg <<'EOF'
  dns_lookup:     %{time_namelookup}\n
  tcp_connect:    %{time_connect}  (pure RTT measurement)\n
  tls_handshake:  %{time_appconnect}\n
  ttfb:           %{time_starttransfer}\n
  total:          %{time_total}\n
EOF

# For continuous monitoring, we wrap this in blackbox_exporter:
# modules:
#   http_2xx_cdn:
#     prober: http
#     timeout: 5s
#     http:
#       preferred_ip_protocol: "ip4"
#       ip_protocol_fallback: false

From Raw RTT to User-Centric Metrics: What to Graph and What to Ignore

Raw RTT measurement is the foundation, but users don't feel RTT, they feel time to first byte and time to first paint. The trick is knowing which metric answers which question. If you're debugging CDN performance for cache efficiency, you want TCP RTT and TLS handshake time isolated from TTFB. If you're answering "is the site fast for users in Jakarta," you want the full TTFB as measured by a real browser, which the folks at web.dev Articles break down beautifully into network vs. server vs. rendering.

We made a rule internally: never present a single "latency" number without its qualifier. Our executive dashboard used to show one big "Global Latency: 34ms" and it was worthless. Now we show three numbers side-by-side and we don't let anyone average them together. It stopped a lot of pointless arguments and, more importantly, it stopped us from optimizing the wrong thing. Chasing a 5ms improvement in backbone RTT doesn't help if your 95th percentile TTFB is dominated by origin cache misses over a congested transit link.

Putting It Into Practice: Building a Trustworthy Synthetic Probe

Alright, enough theory — let's build something you can actually deploy. The biggest mistake I see teams make is reusing their uptime checks for latency measurement. A 200 OK from a 0-byte healthcheck file tells you almost nothing. You want to fetch a real, cacheable object — something between 10KB and 100KB — with cache-busting disabled so you can distinguish HIT from MISS. On a well-tuned edge, a HIT should give you a TTFB of 20-45ms for users within 500km of the PoP, while a MISS that has to go to origin shield will be 120-250ms. If you're only measuring that tiny healthcheck, you'll never see the difference between TCP + TLS setup (which dominates small objects) and actual content delivery.

Your probe needs to break down the waterfall, not just give you a single total time. That means separate timings for DNS lookup, TCP connect, TLS handshake, TTFB, and download. And where you run it matters more than how you code it. Don't just spin up probes in us-east-1 and eu-west-1 — that's measuring cloud-to-cloud, not user experience. You need vantage points inside eyeball networks, residential ISPs, and mobile carriers. We've had good luck mixing bare-metal probes in key metros with lightweight virtual probes for coverage. The tradeoff's real though — virtual appliances add 5-15ms of vSwitch and noisy-neighbor jitter, which is exactly why you need to understand Network Function Virtualization in CDN Infrastructure: Virtual Appliances at Scale before you trust numbers coming from a fully virtualized probe mesh.

Sampling strategy is where most homegrown systems fall apart. Don't hit every PoP every 10 seconds from every probe — you'll create a thundering herd and your own traffic will get rate-limited or prioritized. Stagger probes with jitter, aim for 30-60 second intervals per PoP per probe, and randomize the object URLs across your HIT/MISS pool. We've found that 1 probe per ~2 million users in a region gives you a statistically useful signal without burning cash. And for heaven's sake, disable connection reuse for your latency probes. Keep-alive will hide your TLS and TCP setup costs, which is exactly where 30-40% of your real user latency lives on cold connections.

#!/bin/bash
# cdn-probe.sh - L7 latency breakdown that actually mirrors a browser
# Requires: curl 7.75+, jq

URL="https://cdn.example.com/probe-object-64k.bin"
FORMAT='{"dns":%{time_namelookup},"tcp":%{time_connect},"tls":%{time_appconnect},"ttfb":%{time_starttransfer},"total":%{time_total},"size":%{size_download},"hit":%{header_x-cache}}'

# Force new connection, no keep-alive, no cached DNS
curl -o /dev/null -s -w "$FORMAT\n" \
  --connect-timeout 5 \
  --max-time 10 \
  -H "Cache-Control: no-cache" \
  -H "Pragma: no-cache" \
  --resolve cdn.example.com:443:104.16.12.34 \
  "$URL" | jq .

# For Node.js / Real User Monitoring, use Performance API instead:
# const start = performance.now();
# const res = await fetch(url, {cache: 'no-store', keepalive: false});
# await res.arrayBuffer(); // ensures download time is included

Advanced Patterns and Troubleshooting: When Your Dashboard Lies

You've got probes running, graphs look clean, but users are still complaining. First place I look is Anycast routing. Your probe in Dallas might be hitting your Dallas PoP today and your Chicago PoP tomorrow because of a BGP flap or ECMP rebalance upstream. If you aren't logging the actual PoP ID (from an X-Cache-Status or Server-Timing header) and the TCP server IP for every single measurement, you're flying blind. We've seen anycast shifts add 35ms overnight that looked like "CDN got slower" but was really just telco routing. This is where tight integration with Software-Defined Networking for CDN: Programmable Traffic Control pays off — when your control plane can correlate probe TTFB spikes with SDN-driven path changes in real time, you stop chasing ghosts.

Next culprit is your backbone topology, especially on cache MISSes. A lot of teams obsess over edge latency and forget that a MISS still has to traverse your backbone to shield or origin. If you're on a classic ring topology, a MISS from Miami that has to go to your Ashburn shield might hairpin through Atlanta and add 28ms each way. On a CDN Backbone Topology Design: Ring, Mesh, and Hierarchical Architectures with a proper mesh or hierarchical shield, that same MISS might be 12-18ms faster because it takes a direct peering path. Graph HIT and MISS latency separately — if your p50 HIT is 32ms but p95 MISS is 480ms and your MISS ratio just jumped from 8% to 15% after a purge, your average latency will spike even though edge performance is perfect.

Finally, learn to read the shape of your latency distribution, not just the average. If your median is stable at 45ms but p95 jumps from 110ms to 290ms and p99 goes from 350ms to 800ms, that's almost never your PoP — that's last-mile. Check for bufferbloat on mobile, overloaded CPE, or peering congestion. A dead giveaway is high variance in TTFB but stable TCP connect times. Run parallel MTRs with TCP SYN on port 443, not ICMP — we regularly see ICMP deprioritized by 20-60ms under load, which will make you think the network is congested when it's just ICMP policing. When you see that divergence between TCP connect and ICMP RTT, trust TCP.

Frequently Asked Questions