CDN Performance Testing Methodology: Load Generation, Measurement, and Analysis

Last Black Friday, a client called me at 2 AM because their "blazing fast" CDN was serving product images in 4 seconds flat. Their pre-launch benchmark showed 45ms TTFB — gorgeous numbers, completely useless. They'd tested a single warmed edge PoP with 100 concurrent users pulling the same 10 URLs. Real shoppers hit every PoP at once, requested 50,000 unique SKUs with cold caches, and hammered the origin through a misconfigured shield. That gap between lab and reality is where most CDN performance testing falls apart, and I've seen it sink launches more than any actual CDN failure.

Your Cache Hit Ratio Is Lying to You

When I review a team's first CDN benchmarking results, I can usually spot the problem in ten seconds: a 98% cache hit ratio that no production site ever achieves. They've written a script that loops over a small set of URLs, they run it for an hour to "warm up" the edge, and then they proudly measure performance against a fully primed cache. Real traffic doesn't work that way. You've got cache churn from query string variations, personalized content bypassing cache, purging events, and a long tail of unpopular assets that are always cold. If you don't model that churn, you're not testing your CDN — you're testing your CDN's RAM.

Early in my career, we made this exact mistake with a media customer. We benchmarked video segment delivery and saw sub-30ms edge hits all day. On launch, the origin buckled. Why? We hadn't accounted for the fact that 40% of their library was long-tail content requested once or twice per day per region. Our test had an artificially small working set. Now I always force a realistic cache working set and explicitly control for hit vs. miss. A solid CDN Integration Testing Framework Design: Validating Edge Behavior Programmatically helps here — you need to programmatically verify what's actually a HIT, MISS, or REVALIDATED, not just guess from timing. The HTTP Working Group Specifications on caching semantics are essential reading for understanding how Cache-Control, Vary, and validation headers change that behavior under load.

I don't start a test without defining the cache model first. What percentage of requests *should* be a miss? What’s your expected origin shield efficiency? What's your TTL distribution? Get those numbers from production logs, not assumptions.

# Generate realistic cache footprint - don't just hammer 10 URLs
# This k6 script forces a 60/30/10 distribution: hot / warm / cold objects
import http from 'k6/http';
import { SharedArray } from 'k6/data';

const urls = new SharedArray('urls', function () {
  return JSON.parse(open('./url-manifest.json')); // 50k URLs with popularity weight
});

export let options = {
  scenarios: {
    realistic: {
      executor: 'constant-arrival-rate',
      rate: 2000, // target 2000 RPS
      timeUnit: '1s',
      duration: '30m',
      preAllocatedVUs: 200,
      maxVUs: 1000,
    },
  },
};

export default function () {
  // weighted random selection mimics long-tail access pattern
  const url = urls[Math.floor(Math.pow(Math.random(), 2) * urls.length)];
  // bust cache for 15% of requests to simulate unique query strings / personalization
  const qs = Math.random() < 0.15 ? `?uid=${__VU}-${Date.now()}` : '';
  const res = http.get(`${url}${qs}`, { tags: { cache_expected: qs ? 'MISS' : 'HIT' } });
  // Validate edge cache status, don't just check latency
  if (!['HIT','MISS','REVALIDATED','BYPASS'].includes(res.headers['X-Cache'])) {
    console.warn(`Unexpected cache status: ${res.headers['X-Cache']} for ${url}`);
  }
}

Load Generation That Mirrors Humans, Not Robots

Most load generation tools make it dangerously easy to generate the wrong kind of load. A single beefy VM in us-east-1 blasting an anycast IP doesn't tell you how your CDN behaves when traffic comes from 80 cities at once, over last-mile networks with packet loss, TLS handshakes, and TCP slow start. I've watched tests that looked clean from one region miss global rate-limiting rules and collapsed TCP stacks in Southeast Asia because we simply weren't generating load from there. If your users are global, your load generators must be too.

You also have to decide what you're actually generating: requests per second or concurrent users? They aren't interchangeable. RPS is what stresses the cache and origin; concurrent connections with think time, connection reuse, and browser-like concurrency (6 connections per origin, HTTP/2 multiplexing) is what stresses connection handling and TLS. For CDN benchmarking, I model both. I use arrival-rate executors for pure RPS against the edge, and closed-model VUs with realistic pacing for user-journey simulation that includes HTML, then 30+ assets with dependencies. And don't reuse a single TCP connection forever — real browsers don't. Disable keep-alive for a portion of traffic to force that TLS and TCP handshake cost back into your numbers. That’s where the web.dev Articles on optimizing LCP and understanding connection overhead really ground your test design in how browsers actually fetch.

For anything beyond 10k RPS, you'll need distribution. My rule of thumb: never generate more than 3-5k RPS per generator node, and spread them. This is exactly where a Distributed Load Testing for CDN Edge: Generating Traffic from Multiple Geographies strategy pays off — you coordinate generators in at least 4-6 regions, synchronize time, and stagger ramp-up so you don't accidentally DDoS your own origin shield in the first minute. Don't forget to warm up gradually; I've seen origin autoscaling lag by 3 minutes and completely skew the first 10 minutes of results if you go 0 to 50k RPS instantly.

# wrk2 example for precise RPS with latency correction - run this from 5+ regions
# Don't use wrk without the --rate / --latency fix, you'll get coordinated omission
wrk -t 4 -c 400 -d 15m -R 5000 --latency --timeout 10s \
  -H "User-Agent: Mozilla/5.0 (CDN-PerfTest/1.0)" \
  -H "Accept: text/html,application/xhtml+xml" \
  -s ./scripts/cache-aware.lua \
  https://cdn-test.example.com

-- cache-aware.lua excerpt:
-- request = function()
--   -- 70% HIT path, 20% MISS (cold object), 10% BYPASS (personalized)
--   local r = math.random()
--   if r < 0.7 then path = hot_urls[math.random(#hot_urls)]
--   elseif r < 0.9 then path = cold_urls[math.random(#cold_urls)] .. "?cb=" .. math.random(999999)
--   else path = "/api/user/profile"; headers["Cookie"] = "session=" .. uuid() end
--   return wrk.format("GET", path, headers)
-- end

What to Measure When Averages Hide the Pain

If you give me a report with "average latency: 85ms" I know you haven't found the real problems yet. CDN performance lives and dies in the percentiles and in the relationship between metrics, not in a single number. An average is pulled down by thousands of fast cache hits and completely hides that 5% of users getting 2-second misses. I always measure p50, p90, p95, and p99, split by cache status. A healthy CDN might show p50 HIT at 25ms and p95 MISS at 850ms — that spread is normal and tells you your origin path needs work, not your edge.

You need more than timing, too. Track cache hit ratio per PoP, origin offload percentage, error rates segmented by edge vs. origin, and TLS handshake time. I learned this the hard way when a customer's p95 looked fine but their origin shield hit ratio was 60% — every regional edge was going back to origin instead of the shield, doubling their bill and latency. We only caught it because we were logging X-Cache and X-Shield-Status headers alongside timing. And please, collect client-side data with Navigation Timing and Resource Timing where you can, not just server-side logs. Without that, you can't separate DNS + TCP + TLS + TTFB, and you'll chase the wrong bottleneck.

Before you run the full benchmark, build your environment right. A test against a single origin in a lab doesn't replicate production shielding, WAF, or image optimization. I've had far more reliable results since adopting an End-to-End CDN Testing: Building Production-Like Test Environments approach — mirroring origin infrastructure, using production-identical cache rules, and seeding the test with anonymized production access logs. Otherwise you're just measuring how fast your staging server is.

Putting It Together: A Production-Grade Test Harness in Practice

Look, you can't generate a realistic 50k RPS thundering herd from your MacBook on office Wi-Fi. You'll hit local file descriptor limits and NIC contention long before you stress the edge. A real CDN harness has to be distributed, synchronized, and ruthlessly isolated from your own bottlenecks. We've seen teams celebrate a "CDN failure at 20k RPS" that turned out to be their single t3.medium load generator running out of ephemeral ports.

Here's how we build it for clients: a lightweight orchestrator (usually a control plane on Kubernetes) drives 8-15 generator nodes spread across at least 3 cloud regions and 2 different cloud providers. Each node runs a containerized runner — k6, Vegeta, or Gatling, doesn't matter much — and we sync clocks with NTP/PTP so we can correlate logs to the millisecond. Critically, we cap each generator at 60-70% CPU. If a generator pegs CPU, its latency measurements are garbage. We also pre-warm the dataset. Don't start measuring until you've run at 20% of target load for 5 minutes to populate the edge and let anycast routing settle.

Before you declare victory, you need a clean baseline. Run a 10-minute control test with origin bypass disabled, then another with a cache-busting query string to measure pure origin + CDN pass-through. That delta is your actual cache benefit. If you're not doing this, you're guessing. And for any test involving cache correctness — like purging, tiering, or Vary headers — you’ll want to layer in dedicated CDN Cache Validation Testing: Verifying Correct Caching Behavior Across Edges so you know a 99% hit ratio isn't just serving stale private data.

// k6 example: realistic CDN test with Zipf-like asset distribution and cache-busters
import http from 'k6/http';
import { sleep } from 'k6';

export const options = {
  scenarios: {
    realistic_browsing: {
      executor: 'ramping-arrival-rate',
      startRate: 500,
      timeUnit: '1s',
      preAllocatedVUs: 2000,
      stages: [
        { target: 8000, duration: '5m' },  // warm-up, don't measure
        { target: 25000, duration: '10m' }, // sustained load
        { target: 40000, duration: '2m' },  // spike / thundering herd
      ],
    },
  },
  thresholds: {
    'http_req_duration{p95}': ['p(95)<120'], // p95 under 120ms on cache HIT
    'http_req_failed': ['rate<0.01'],        // <1% errors even during origin collapse
  },
};

const ASSETS = JSON.parse(open('./manifest-zipf.json')); // top 10% assets = 85% of requests

export default function () {
  const asset = ASSETS[Math.floor(Math.random() * ASSETS.length)];
  // 95% HIT path, 5% MISS/bust to test origin shielding
  const url = Math.random() > 0.05 ? asset.url : `${asset.url}?cb=${Date.now()}-${__VU}`;
  const res = http.get(url, { headers: { 'Accept-Encoding': 'gzip, br' } });
  sleep(Math.random() * 2 + 0.5); // think time: 0.5s - 2.5s
}

Advanced Patterns and Troubleshooting When Nothing Makes Sense

If your p50 drops but your p99 explodes to 1.8s during a spike, you haven't found a CDN edge problem — you've usually found origin collapse or request coalescing gone wrong. We had a retail client whose hit ratio stayed at 94% during Black Friday simulation, yet origin hit 100% CPU. The culprit? 6,000 concurrent cache MISSes for the same just-purged product JSON. Without origin shielding or stale-while-revalidate, every edge fetched in parallel. You’ve got to test that collapsed-forwarding behavior explicitly. Ramp to zero, purge a hot key, then hit it with 10k RPS in the next 500ms. If your origin sees more than 3-5 requests for that key, your shielding is misconfigured.

Another classic head-scratcher: results that swing 30-40% between runs with identical configs. That’s almost always noisy neighbors, anycast churn, or tiered cache inconsistency. Don't average three runs and call it a day. Tag every request with a unique X-Test-ID and log CF-Cache-Status, Age, X-Cache, and PoP ID. You'll often see that 22% of your MISSes were all routed to a single cold PoP in São Paulo because your load generator placement was biased. Fix it by pinning generators to specific geos and testing tiered cache vs. flat. A proper tiered setup should reduce origin egress by 60-80% on a 85%+ hit ratio workload — if it doesn't, your mid-tier isn't actually shielding.

And when you need to know what happens when things *really* break, performance testing alone won't save you. You have to combine it with failure injection. What happens to TTFB when you kill an origin region mid-test? Does failover add 80ms or 800ms? We routinely pair load generation with Chaos Testing at CDN Edge: Injecting Faults to Validate Resilience and scripted CDN Failover Testing Playbook: Scripted Scenarios for Disaster Recovery Validation runs. Last quarter, that combo exposed a 45-second failover black hole where the CDN kept trying a dead origin because health checks were set to 30s intervals. You'd never catch that with steady-state load alone.

Frequently Asked Questions