RTT Measurement Methodology for CDN: Getting Accurate Round-Trip Times

Last spring we rolled out a new PoP in Jakarta and the dashboard lit up green — average RTT dropped from 85ms to 22ms overnight. The product team celebrated, but our real-user metrics barely moved. Turns out we'd switched our health checks from TCP handshakes to ICMP pings without updating the graph legend, and we were congratulating ourselves on a measurement artifact. If you've spent any time tuning a CDN, you've seen some version of that story.

The RTT You're Actually Measuring Isn't What Users Feel

Most folks start their RTT measurement journey with ping, and I get it. It's simple, it's everywhere, and it gives you a number in milliseconds that feels authoritative. The problem is that ICMP round-trip time has almost nothing to do with the round-trip time a browser experiences when it fetches an object through your CDN. Routers often deprioritize or rate-limit ICMP, anycast routing can send your ping one way and your HTTP traffic another, and you aren't measuring TCP stack behavior at all. When I was debugging slow TTFB for an e-commerce client in São Paulo back in 2019, their synthetic pings showed a crisp 12ms to our edge, but the TCP three-way handshake was consistently taking 45ms because of a congested peering link that only affected TCP.

For CDN work, what you really care about is the transport-layer RTT — how long it takes for a SYN to get a SYN-ACK back, or for a data segment to be acknowledged. That number directly drives your congestion window growth, your retransmit timers, and ultimately your throughput. It’s also what the kernel actually tracks. Modern Linux exposes this through TCP_INFO, and if you aren't pulling from there, you're guessing. The HTTP Working Group Specifications are pretty clear that HTTP semantics sit on top of this transport reality, so you can't reason about HTTP/2 or HTTP/3 performance without a solid transport measurement first. And once you have that true TCP RTT, you'll quickly see why TCP Window Scaling and Its Impact on CDN Throughput: Why Defaults Fall Short matters so much — without proper scaling, a 40ms RTT caps you long before bandwidth does.

We learned to stop trusting any single vantage point. A single probe from Virginia to Singapore tells you about that path at that second, not about what your users get. You need to capture RTT at the edge, passively, for every connection, and timestamp it as close to the wire as possible. That means pulling from the kernel, not the application log. Here's the kind of one-liner we still use on our edge nodes to sanity-check what the kernel thinks the RTT is before we trust any higher-level dashboard:

# Check real TCP RTT as seen by the kernel (requires iproute2)
# srtt is smoothed RTT in milliseconds, rttvar is variance
ss -ti '( dport = :443 )' | grep -E 'rtt:|srtt'

# Example output for a single client connection:
#  cubic wscale:7,7 rto:204 rtt:38.421/5.212 mss:1428 pmtu:1500
#  ^ rtt is 38.4ms with 5.2ms variance - this is your ground truth
#  Compare this to your application-level timing, not ping

# For continuous collection via getsockopt(TCP_INFO)
# tcpi_rtt and tcpi_rttvar are in microseconds
cat /proc/net/tcp | awk '{print $10}' | head

Active Probes vs. Passive Observation: Picking the Right Measurement Methodology

There's a classic split in our field between active and passive RTT measurement, and I've found most teams pick one and ignore the other — which hurts their latency accuracy. Active measurement is you sending probes on purpose: synthetic HTTP requests, TCP SYN pings with hping3, or RIPE Atlas credits. It's great because you control the timing, the location, and the protocol. We run active probes every 30 seconds from every PoP to every other PoP to build a latency mesh. It tells us when a transit provider is having a bad day before customers notice. But active probes don't see real user paths, real congestion, or real last-mile quirks like bufferbloat on a home router.

Passive observation flips that. Instead of generating traffic, you watch real user traffic and extract the RTT from it. The kernel already does this for every TCP connection — it measures the time between sending a segment and getting its ACK. You can export it from eBPF, from NGINX's $tcpinfo_rtt variable, or from Varnish/ATS logs. This is how we feed our percentile dashboards, and it's far more representative. The trade-off is noise: retransmits, delayed ACKs, and GRO/LRO batching can all distort a naive measurement. I remember a case where our passive metrics spiked to 300ms every night at 8pm in Mumbai — not because the network got worse, but because a popular set-top box was using a 200ms delayed ACK timer. If we'd only looked at averages, we'd have blamed the network.

If you're building this for the first time, don't try to invent a new collector. Start with what's already in your proxy and augment it. We log tcpi_rtt for the initial handshake and then sample it again after 1MB of transfer to catch congestion-driven inflation. That pair tells you a lot — handshake RTT is your clean propagation delay, while the later sample includes queuing. Understanding that difference is key to Bandwidth-Delay Product and CDN Tuning: Filling the Pipe Without Overwhelming It, because BDP math falls apart if you plug in the wrong RTT. Here's a minimal NGINX config we use to expose kernel RTT without adding sidecar agents:

# nginx.conf - log true kernel RTT per request
# Requires nginx with --with-http_realip_module and recent kernel
log_format rtt_log '$remote_addr - $request '
                   'rtt=$tcpinfo_rtt rttvar=$tcpinfo_rttvar '
                   'cwnd=$tcpinfo_snd_cwnd status=$status';

server {
    listen 443 ssl http2;
    access_log /var/log/nginx/rtt_access.log rtt_log;

    location / {
        # For QUIC/HTTP3, use $quic_rtt if available (different stack)
        add_header X-Edge-RTT $tcpinfo_rtt always;
        proxy_pass http://origin;
    }
}

# Validate in real time:
# tail -f /var/log/nginx/rtt_access.log | awk '{if ($6 > 100000) print}'
# (tcpinfo_rtt is in microseconds - 100000 = 100ms)

Scrubbing the Signal: Filtering Outliers for Real Latency Accuracy

Raw RTT data is messy, and if you just average it, you're lying to yourself. We've seen single PoPs ingest 2 billion RTT samples a day, and a handful of pathological clients will wreck your mean. Satellite users with 600ms+ RTT, clients behind retransmitting middleboxes, or connections that had a single tail loss and waited a full RTO — they all inflate the average without telling you anything about the experience of 95% of your users. That's why our measurement methodology has a filtering stage before anything hits a graph. We drop samples where tcpi_retrans > 0, where the RTT sample was taken during loss recovery, and where the socket's RTT variance is more than 3x the smoothed value. The web.dev Articles on optimizing Core Web Vitals make a similar point: you can't optimize what you can't measure cleanly.

Once you've filtered, don't look at the mean. Seriously. For CDN performance science, the distribution tells the story. Your p50 tells you about your typical user path, your p95 tells you about congestion and peering health, and your p99 often reveals client-side problems you can't fix but need to understand. We had a stretch where p50 was a beautiful 18ms in London, but p95 was 210ms — turned out one ISP was routing 12% of its users over a saturated exchange. The average hid it completely. If you want to get better at reading those curves, CDN Percentile Latency Analysis: What p50, p95, and p99 Actually Tell You breaks down why those three numbers behave so differently on a global edge.

Below is the comparison we show to new engineers during onboarding to hammer home that method matters more than math. All four rows were collected in the same hour, from the same PoP, to the same set of client prefixes — only the method changed.

Putting It Into Practice: Passive RTT Collection at the Edge

We've talked theory long enough — here's how we actually collect RTT without melting your edge. Don't write a userspace ping loop, your kernel's already tracking it for you. Every established TCP socket maintains a smoothed RTT (srtt) estimate for its retransmission logic, and you can pull it for free. On Linux, that's getsockopt(TCP_INFO) or reading tcp_info.tcpi_rtt directly. It's updated on every ACK, it's in microseconds, and it costs you essentially nothing compared to injecting synthetic probes at 10k RPS per PoP.

The trick isn't getting the number, it's where and how often you sample it. You don't want to log it on every request — at 150k-200k req/s per edge node, that'll drown your telemetry pipeline. We typically sample 1:500 or 1:1000 at the connection close, and we do it via eBPF so we never touch the hot path. Attach to tcp_rcv_established or just periodically scrape sock->srtt_us. You'll see edge RTTs cluster around 12-35ms for broadband users in-region, but don't average them. Export a histogram or at least p50, p75, p95, p99. A mean RTT of 28ms means nothing if your p95 is 180ms because of bufferbloat on last-mile CPEs.

Once you've got it, ship it with the right dimensions. Don't just emit a global rtt_ms. Tag it by PoP, ASN, /24, and protocol (TCP vs QUIC). QUIC's RTT will consistently read 3-8ms higher than TCP on the same path because of userspace ACK handling, and that's expected. If you're not separating them, you're chasing ghosts. We've had great luck pushing this as an OpenTelemetry histogram to Prometheus/VictoriaMetrics — cardinality is manageable if you aggregate client prefixes at the edge before export.

# Passive TCP RTT extraction via eBPF - low-overhead per-socket sampling
# Requires bcc / libbpf. Reads tcp_sock->srtt_us (scaled by 8 in kernel)
import bpf

BPF_PROGRAM = r"""
#include <net/sock.h>
#include <net/tcp.h>

BPF_HISTOGRAM(rtt_hist, int, 64);
BPF_HASH(rtt_by_asn, u32, u64);

int trace_tcp_rtt(struct pt_regs *ctx, struct sock *sk) {
    struct tcp_sock *tp = (struct tcp_sock *)sk;
    // srtt_us is stored as srtt >> 3, convert to ms
    u32 srtt_ms = tp->srtt_us >> 3;
    if (srtt_ms == 0 || srtt_ms > 3000) return 0; // filter unsampled / insane
    
    srtt_ms = srtt_ms / 1000;
    rtt_hist.increment(bpf_log2l(srtt_ms));
    return 0;
}
"""
# Attach to tcp_rcv_established, sample 1:1000 via BPF sampling logic
# Export as prometheus histogram with labels: pop, asn, proto=tcp

When Your RTT Data Lies: Debugging Noisy and Misleading Signals

If your dashboard suddenly shows RTT doubling overnight but your origin latency hasn't budged, don't panic and reroute traffic just yet. Nine times out of ten, it's not the network — it's what you're measuring. TLS handshakes are the classic trap. A full TLS 1.3 handshake adds 1-RTT, and if you're measuring from SYN to HTTP TTFB, you're lumping crypto into network latency. We've seen customers report 110ms "RTT" that was actually 35ms network + 75ms handshake on cold connections. Switch to measuring only on resumed sessions or after the handshake, or better yet, use TCP_INFO's srtt which is handshake-agnostic. For QUIC, rely on the QUIC stack's own latest_rtt and smoothed_rtt — don't try to infer it from HTTP timing.