Distributed Load Testing for CDN Edge: Generating Traffic from Multiple Geographies

We shipped a major live event for a sports broadcaster last year and our single-region CDN stress test looked beautiful — 50k RPS from us-east-1, sub-40ms TTFB, zero errors. Then the actual match started and viewers in São Paulo and Singapore couldn’t get a segment without rebuffering. I’ve seen that failure mode more times than I can count over the last decade: you don’t test where your users are, you don’t know how your edge really behaves. A proper distributed load test fixes that blind spot before it costs you an audience.

Why Your Single-Origin Test Masks Edge Failures

When you fire all your load from one AWS region or your office network, you’re only exercising one or two CDN PoPs and one path to origin. It feels efficient, but it hides the problems that actually hurt in production. Anycast routing, regional cache fill, TLS handshake costs on long-haul connections, and origin shield congestion — none of those show up when 95% of your traffic hits Ashburn. I’ve watched teams celebrate a clean test report on Friday and then spend Saturday firefighting 503s that only appeared in Mumbai and Frankfurt.

A true multi-geo testing approach forces every layer to do its job. You’re validating that your cache-key design shards correctly, that stale-while-revalidate works under thundering herd from cold PoPs, and that your origin can handle ten different shields filling at once instead of one. If you’re building out your automation, it helps to pair this with a solid CDN Integration Testing Framework Design: Validating Edge Behavior Programmatically so you’re not just hammering the edge, you’re asserting on headers, status codes, and cache states from each geography.

The other thing folks miss is DNS and connection reuse. Real users don’t share a single resolver or a warm connection pool. Distributed generators naturally create the DNS lookup variance and the mix of HTTP/2 and HTTP/3 handshakes you’ll see globally. Without that, your edge load test is essentially a best-case fantasy. The HTTP Working Group Specifications are pretty clear on how connection coalescing and prioritization should work, but you’ll never verify it with synthetic traffic from one location.

# k6 script configured for distributed execution
# Run coordinators in 4 regions, each reports to central influxdb
import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  scenarios: {
    edge_warmup: {
      executor: 'ramping-arrival-rate',
      startRate: 500,
      timeUnit: '1s',
      preAllocatedVUs: 200,
      maxVUs: 1000,
      stages: [
        { target: 2000, duration: '5m' }, // ramp per region
        { target: 5000, duration: '10m' }, // sustained CDN stress test
        { target: 0, duration: '2m' },
      ],
      exec: 'edgeTest',
    },
  },
  thresholds: {
    'http_req_failed': ['rate<0.01'],
    'http_req_duration{cache:hit}': ['p(95)<80'],
    'http_req_duration{cache:miss}': ['p(95)<250'],
  },
};

export function edgeTest() {
  // geo tag is injected via env var on each generator node
  const geo = __ENV.GEO || 'unknown';
  const res = http.get('https://vod.example.com/segment/1080p/init.mp4', {
    tags: { geo: geo, cache: res.headers['X-Cache'] },
  });
  check(res, {
    'is 200': (r) => r.status === 200,
    'cache hit or miss': (r) => /HIT|MISS/.test(r.headers['X-Cache']),
  });
  sleep(Math.random() * 0.5);
}

Building a Multi-Region Traffic Mesh That Actually Works

I don’t recommend trying to spin up 50 global generators on day one. Start with four to six regions that mirror your real audience — for most of my clients that’s us-east-1, us-west-2, eu-west-1, ap-southeast-1, sa-east-1, and ap-northeast-1. I’ve had better luck with a handful of well-placed cloud VMs than with trying to manage a fleet of spot instances across every provider. You want consistent CPU and network, not the cheapest box that throttles under load. We use Terraform to stamp out identical generator stacks and a lightweight coordinator to sync start times with NTP; even a two-second skew can ruin your ramp analysis.

Synchronization is where most distributed load test setups fall apart. If your generators start at different times, you can’t tell if a latency spike is from edge saturation or just one region lagging behind. We run a simple control plane that pushes the test plan, checks health, and then broadcasts a “go” timestamp 60 seconds in the future. Each node also tags every request with its region and a unique run ID in a custom header, so we can slice logs at the CDN and origin without guessing. You’ll want that traceability when you’re trying to explain why End-to-End CDN Testing: Building Production-Like Test Environments matters — production-like means geographically distributed, not just data-identical.

Don’t forget egress costs and rate limits. I once burned $3,000 in a weekend because we left generators pulling 4K segments at origin-miss on loop. Set byte caps per node, use range requests or small object variants for pure connection testing, and verify your CDN’s per-PoP rate limiting won’t block your own test. Articles on web.dev Articles do a good job explaining how connection and resource prioritization affect perceived performance, and that’s exactly what you’re trying to simulate — not just raw bytes, but realistic browser behavior from different networks.

# coordinator.sh - sync start across all geo generators
#!/bin/bash
set -euo pipefail
REGIONS=("us-east-1" "eu-west-1" "ap-southeast-1" "sa-east-1")
RUN_ID="edge-test-$(date +%Y%m%d-%H%M%S)"
START_AT=$(date -d "+90 seconds" -u +%Y-%m-%dT%H:%M:%SZ)

echo "Preparing run $RUN_ID for $START_AT"

for REGION in "${REGIONS[@]}"; do
  echo "Pushing config to $REGION..."
  ssh "generator-$REGION" \
    "cat > /tmp/k6-config.json <<EOF
    {\"runId\": \"$RUN_ID\", \"startAt\": \"$START_AT\", \"geo\": \"$REGION\"}
EOF"
  # pre-flight check: clock sync and connectivity
  ssh "generator-$REGION" "chronyc tracking && curl -I -s https://vod.example.com/health | head -1"
done

echo "Broadcasting GO. All nodes will start at $START_AT"
for REGION in "${REGIONS[@]}"; do
  ssh "generator-$REGION" "nohup k6 run --env GEO=$REGION --env RUN_ID=$RUN_ID /opt/tests/edge-script.js > /tmp/k6-$RUN_ID.log 2>&1 &" &
done
wait
echo "All regions launched. Tail logs with: ./collect-logs.sh $RUN_ID"

What to Measure When Every Edge Is Under Load

You can’t just watch average latency. For an edge load test you need to separate hit and miss paths per PoP, otherwise a 90% hit ratio in Virginia hides a 40% hit ratio in Sydney. I always graph four things by geography: TTFB, cache hit ratio, origin egress by shield, and error rate (especially 502/503/530). If your CDN provides Server-Timing or X-Cache-Status, use it — that’s how you prove a request actually exercised the edge and didn’t just get served from a single hot node.

Thresholds should be per-region, not global. It’s completely normal for p95 from São Paulo to be 80ms higher than from London, even when healthy. What you’re looking for is divergence during ramp: if ap-southeast-1’s p95 doubles while eu-west-1 stays flat, you’ve found a capacity or routing issue. Our methodology doc goes deeper on this — see CDN Performance Testing Methodology: Load Generation, Measurement, and Analysis for how we set baselines and compare runs. I’ve learned to keep the pass/fail criteria in version control alongside the test scripts so we don’t quietly move the goalposts after a bad run.

Here’s a snapshot from a recent 4-region, 20k RPS per region test we ran for a streaming customer. We held 80k RPS globally for 15 minutes after a cold start. The table below shows why aggregating metrics globally would have lied — overall hit ratio looked fine at 88%, but ap-southeast-1 was struggling.

Putting It Together: A Production-Grade Global Load Rig

You don't need a bespoke platform to do this right. We've had great luck with a boring, repeatable stack: containerized k6 or Vegeta runners orchestrated via Terraform, deployed as spot instances in 6-8 regions simultaneously. Don't try to run 100k RPS from a single t3.micro in us-east-1 and call it "global" — you'll hit NIC limits long before you stress the edge. We typically size for 15k-20k RPS per generator node (c6i.large is our sweet spot at ~$0.08/hr spot), and we scale horizontally. For a full 120k RPS global test, that's 8 nodes across ap-southeast-1, eu-west-1, us-west-2, sa-east-1 and friends, all firing at once.

The trick isn't just generating load, it's generating realistic load. If every request has a unique cache-busting query string, you'll hammer origin and learn nothing about edge caching. If every request is identical, you'll get a 98% cache-hit ratio that hides your origin shielding problems. We aim for a 70/30 split — 70% hot, cacheable assets (with proper If-None-Match and ETag handling) and 30% dynamic/uncacheable API calls. And if you haven't already, get your CDN Cache Validation Testing: Verifying Correct Caching Behavior Across Edges pipeline solid first — there's no point load testing an edge that isn't even caching correctly.

Here's the skeleton we use to orchestrate a synchronized burst. It’s not fancy, but it guarantees all regions start within a 2-second window and report back to a central VictoriaMetrics bucket. We've seen folks lose 4+ hours debugging "regional latency" that was just staggered start times.

#!/bin/bash
# global-edge-burst.sh - synchronized multi-region k6 run
REGIONS=("us-east-1" "eu-west-1" "ap-northeast-1" "ap-southeast-2" "sa-east-1" "us-west-2")
TEST_ID="edge-load-$(date +%s)"
DURATION="10m"
TARGET_RPS=15000  # per region, so ~90k RPS global

for REGION in "${REGIONS[@]}"; do
  echo "[$REGION] Launching generator..."
  aws ec2 run-instances \
    --region $REGION \
    --image-id ami-0c55b159cbfafe1f0 \
    --instance-type c6i.large \
    --user-data "#!/bin/bash
      docker run --rm -e TEST_ID=$TEST_ID \
        -v /opt/k6/scripts:/scripts \
        grafana/k6 run /scripts/edge-mixed-workload.js \
        --vus 1500 --duration $DURATION \
        --rps $TARGET_RPS --out influxdb=http://metrics.internal:8086/k6" &
done
wait
echo "All regions firing. Check Grafana dashboard: $TEST_ID"
# Tear down with: aws ec2 terminate-instances --instance-ids $(cat /tmp/ids)

Advanced Patterns: When Your Edges Start Lying to You

Once you're past the basics, you'll hit the weird stuff — and it won't show up in your aggregate dashboard. My favorite example: we ran a 50k RPS test and saw a beautiful 42ms p95 globally, but p95 in ap-southeast-1 was 340ms with a 1.8% error rate. The global average completely masked it. Turns out Anycast was routing 30% of Singapore traffic to our Los Angeles PoP due to a BGP flap we didn't catch. If you aren't slicing metrics by PoP (not just region) and by cache status (HIT vs. MISS vs. REVALIDATED), you're flying blind. Don't trust the CDN's own PoP header blindly either — we've seen stale X-Cache headers lie during overload. Correlate with your own edge logs.

Another gotcha is coordinating faults with load. A clean load test tells you what happens when everything works; it tells you nothing about what happens when an edge fills its disk or an origin times out. That's where you need to start pairing load with failure. We usually schedule a follow-up run that injects latency and 5xx errors at the origin while the mesh is still at 80% load. If you want a structured way to do that, our approach to Chaos Testing at CDN Edge: Injecting Faults to Validate Resilience covers the fault types that actually break edge logic — like slow-read attacks and origin connection pool exhaustion. Start with a 5-second origin delay injection; you'd be surprised how many edges will hold connections for 30+ seconds and OOM.

Finally, have a kill switch and a rollback playbook before you start. We've seen a team accidentally DDoS their own origin shield in us-east-1 because their "cache-hit" test was actually generating 60k origin fetches per second after a bad regex invalidated everything. Set hard caps: alert if origin QPS exceeds 10k for more than 60 seconds, and auto-terminate generators if edge 5xx exceeds 0.5% for 2 minutes. For anything beyond ad-hoc testing, formalize it with a CDN Failover Testing Playbook: Scripted Scenarios for Disaster Recovery Validation so you're not improvising when traffic needs to actually fail over from eu-west-1 to eu-central-1 at 2 a.m. You'll thank yourself when it's not a drill.

Frequently Asked Questions