A/B Performance Testing for CDN Configurations: Measuring Real Impact

A few years ago we pushed a "simple" Brotli level change to our edge and watched synthetic monitors turn green across the board. Twenty-four hours later, real users in Southeast Asia were seeing 400ms slower Largest Contentful Paints and we didn't catch it until support tickets piled up. That incident taught me that CDN configuration changes don't have a single performance number — they have a distribution, and you can't trust a single-point comparison. If you're not running a proper A/B performance comparison on live traffic, you're guessing.

Why Spot Checks and Synthetic Tests Miss the Real Story

Synthetic monitoring has its place, but I've learned not to trust it for CDN testing. It hits from a handful of well-connected data centers, reuses warm TCP connections, and often bypasses the exact cache states real users hit. When we tested a new tiered caching topology last year, our Catchpoint nodes showed a clean 18% TTFB win. Real user data told a different story: cache hit ratio improved for North America but got worse for long-tail assets in Europe because our shield POP was suddenly overloaded. You wouldn't see that without splitting live traffic.

The other trap is testing sequentially — run config A for a week, then config B for a week, and compare. That doesn't work either. Internet weather changes, origin load changes, even major sporting events skew traffic patterns. I made that mistake in 2019 comparing two TCP tuning profiles a week apart and declared a winner that disappeared when we reversed the order. A true A/B test runs both configurations simultaneously on randomized, comparable user populations. That way, when you measure configuration impact, you're isolating your change from everything else happening on the network.

Understanding what's actually on the wire helps here. Small changes in how you handle connection reuse or compression can interact with Bandwidth-Delay Product and CDN Tuning: Filling the Pipe Without Overwhelming It and TCP Window Scaling and Its Impact on CDN Throughput: Why Defaults Fall Short in ways that synthetics will never surface. For accurate baseline data, you also need solid timing foundations as described in RTT Measurement Methodology for CDN: Getting Accurate Round-Trip Times. The HTTP semantics themselves haven't changed — see the HTTP Working Group Specifications — but how you apply them at the edge changes everything.

// Edge-based traffic split with sticky assignment (Cloudflare Workers example)
// Assigns users to control/variant and keeps them there for the experiment

export default {
  async fetch(request, env) {
    const COOKIE_NAME = "cdn_ab";
    const cookie = request.headers.get("Cookie") || "";
    let variant = cookie.match(/cdn_ab=(control|variant)/)?.[1];

    if (!variant) {
      // 50/50 split using crypto random - don't use Math.random() for this
      const rand = crypto.getRandomValues(new Uint8Array(1))[0] / 255;
      variant = rand < 0.5 ? "control" : "variant";
    }

    // Forward variant as header to origin and for logging
    const newHeaders = new Headers(request.headers);
    newHeaders.set("X-CDN-AB-Variant", variant);
    
    let response = await fetch(request, { headers: newHeaders });
    
    // Stickiness: set cookie if not present
    if (!cookie.includes(COOKIE_NAME)) {
      response = new Response(response.body, response);
      response.headers.append(
        "Set-Cookie", 
        `${COOKIE_NAME}=${variant}; Path=/; Max-Age=86400; SameSite=Lax`
      );
    }
    
    // Add variant to Server-Timing for RUM correlation
    response.headers.append("Server-Timing", `ab;desc="${variant}"`);
    return response;
  }
}

Designing a Clean Split: Randomization, Stickiness, and Sample Size

Getting the split right is where most teams trip. You need three things: proper randomization, sticky assignment, and enough traffic. Randomization should happen at the edge, as close to the user as possible, and it should be based on a stable identifier. I prefer a cookie with a header fallback rather than IP hashing — carrier-grade NAT and shared corporate egress IPs will otherwise cluster thousands of users into the same bucket and bias your results. Whatever you do, don't randomize per-request. If a single user bounces between control and variant during a session, you'll pollute your navigation timing data and you can't attribute performance changes cleanly.

Stickiness also matters for cache efficiency. If you don't pin users, you'll effectively double your cache working set and artificially depress hit ratios for both variants. We learned this the hard way when a non-sticky test made both variants look 8% worse than production. Once we pinned with a cookie and allowed the cache key to include the variant only where the config actually differed (like compression level or `Vary` handling), hit ratios normalized and the true difference appeared. If your change doesn't affect cacheability, don't vary the cache key at all — you'll just create unnecessary fragmentation.

Before you start, run a power calculation. You can't just say "we'll run it for two days and see." For tail metrics like p95 TTFB, you'll need far more samples than for median. Web performance data is noisy, and as the web.dev Articles on Core Web Vitals emphasise, real-user distributions are long-tailed. I usually estimate required sample size offline so I don't have to peek early and fool myself.

# Sample size estimation for p95 comparison (Python)
# Use this BEFORE you start the test - don't peek and stop early

import statsmodels.stats.api as sms
import numpy as np

# Inputs from historical RUM data
baseline_p95 = 820  # ms, TTFB p95
minimum_detectable_effect = 40  # ms - what's meaningful for your users?
std_dev = 310  # ms, from last 7 days of RUM
alpha = 0.05  # false positive rate
power = 0.80  # 80% chance to detect real effect

effect_size = minimum_detectable_effect / std_dev
required_n = sms.NormalIndPower().solve_power(
    effect_size=effect_size, 
    alpha=alpha, 
    power=power, 
    alternative='two-sided'
)

print(f"Samples needed per variant: {int(required_n)}")
# Example output: ~945 per variant for mean comparison
# For p95, multiply by ~3-4x or use bootstrap simulation

# Quick bootstrap check for percentiles
def bootstrap_p95_ci(data, n_boot=5000):
    boot = [np.percentile(np.random.choice(data, len(data)), 95) 
            for _ in range(n_boot)]
    return np.percentile(boot, [2.5, 97.5])

Metrics That Reveal Actual Configuration Impact

Don't stare at averages. For CDN performance science, you want the full distribution and a few tightly chosen metrics that connect directly to what your config changed. If you tuned compression or minification, look at download size and LCP. If you changed cache rules, origin offload and hit ratio are your primary signals, but you still need to confirm they translate to user-perceived improvement. My default dashboard for any CDN A/B test includes TTFB, first-byte tail latencies (p75, p95), cache hit ratio, origin request rate, and error ratio.

Split those metrics by geography and network quality. A global average hides everything. I've seen Brotli-11 win on fiber in Germany but lose badly on 3G in India because decompression time wiped out the byte savings. Break down by continent, connection type (via Effective Connection Type or RTT bucket), and cache status (HIT vs MISS). A configuration that improves HITs but makes MISSes substantially slower can still hurt your p95 if your hit ratio isn't high enough to compensate.

Here's what a real result table looks like after 72 hours with ~4.2M requests per variant on a media site where we tested increasing `origin_keepalive_timeout` and enabling TCP window scaling tuning. The aggregate "performance comparison" looked flat until we segmented.

Implementing the Split in Production: Edge Logic That Won't Nuke Your Cache

We’ve all seen what happens when someone implements an A/B test by adding a random header at the origin — you instantly fragment your cache, your hit ratio tanks by 20-30%, and your "performance test" is really just measuring how bad your origin is under load. If you’re going to split traffic at the CDN layer, you’ve got to do it at the edge, before the cache lookup, and you need to be deliberate about how you key it.

The cleanest approach we’ve used in production is a deterministic hash on a stable visitor identifier, combined with a sticky cookie for returning users. Don’t hash on IP alone — carrier-grade NAT and mobile networks will cluster thousands of users into a single bucket and skew your results. Instead, hash a combination of IP + User-Agent class or better yet, a first-party device ID if you have one. Assign the bucket, set a cookie like cdn_ab=config_b; Path=/; Max-Age=2592000, and — this is critical — include that bucket value in your cache key variation or use Vary correctly. On Fastly/Varnish this means adding the group to hash_data, on CloudFront it’s a cache policy that whitelists the cookie, on Cloudflare Workers you handle it manually. You want two logical cache namespaces, not one poisoned cache.

Keep your split at 50/50 if you’re testing a pure performance config, or 90/10 if you’re nervous about risk. But don’t eyeball it for a day and call it. You’ll need at least 100k-250k requests per variant and a full weekly traffic cycle — we typically run for 7 to 14 days minimum — to smooth out diurnal patterns and any peering blips. And log everything: request ID, PoP, assigned group, cache status (HIT/MISS/PASS), and your core metrics. If you can’t join your RUM data back to the A/B assignment later, you’re flying blind.

// Edge Worker (Cloudflare / Fastly Compute@Edge) - Sticky A/B with cache-safe variation
const VARIANTS = { control: 50, variant: 50 }; // 50/50 split

function getBucket(request) {
  // 1. Check stickiness first
  const cookie = request.headers.get("Cookie") || "";
  const match = cookie.match(/cdn_ab=([^;]+)/);
  if (match) return match[1];

  // 2. Deterministic hash for new visitors (FNV-1a on IP + UA class)
  const id = request.headers.get("CF-Connecting-IP") + "|" + getUAClass(request.headers.get("User-Agent"));
  let hash = 2166136261;
  for (let i = 0; i < id.length; i++) {
    hash ^= id.charCodeAt(i);
    hash = Math.imul(hash, 16777619);
  }
  const pct = Math.abs(hash) % 100;
  return pct < VARIANTS.control ? "control" : "variant";
}

export async function handleRequest(request) {
  const bucket = getBucket(request);
  // Add to cache key and origin header
  request.headers.set("X-CDN-AB", bucket);
  let response = await fetch(request, {
    cf: { cacheKey: `${request.url}::ab=${bucket}` }
  });
  // Set sticky cookie if missing
  if (!request.headers.get("Cookie")?.includes("cdn_ab=")) {
    response = new Response(response.body, response);
    response.headers.append("Set-Cookie", `cdn_ab=${bucket}; Path=/; Max-Age=2592000; SameSite=Lax`);
  }
  response.headers.set("X-CDN-AB-Group", bucket);
  return response;
}

Advanced Patterns and Troubleshooting When Your Data Looks Wrong

If your p50 drops by 15ms but your p95 jumps by 40ms, you don’t have a winner — you have a distribution problem. This is exactly why we never look at averages in isolation. Averages hide tail behavior, and CDN config changes almost always show up in the tail first. When we rolled out tiered shielding with a new TCP congestion profile, our mean TTFB looked flat, but CDN Percentile Latency Analysis: What p50, p95, and p99 Actually Tell You made it obvious: p50 improved by 8ms, p95 regressed by 22ms for users on lossy mobile networks. You need to segment your results by percentile, by PoP, by cache status, and by network type (wired vs. mobile) before you declare victory.

The other killer is noisy data you forgot to filter. Bots are the obvious one — we’ve seen tests where 12% of "variant" traffic was actually Googlebot and Ahrefs crawling from a single AWS region, completely skewing origin latency. Filter known bot ASNs and user-agents before analysis. Then check for CDN Jitter Measurement and Its Impact on User Experience — if your variant shows higher jitter even with a slightly lower median latency, users on video or real-time calls will feel it as stutter. We flag any variant that increases jitter by more than 3-4ms at p95 as a regression, even if throughput looks better. And don’t ignore the low-level network stuff: we once chased a 5% revalidation slowdown for two weeks before realizing a seemingly harmless header-order change was causing Packet Reordering Effects on CDN Performance: The Hidden Throughput Killer on one transit provider, hammering throughput for 18% of EU users.

For high-stakes configs, use a holdout pattern. Instead of a clean 50/50 flip, run a 45/45/10 split — 45% control, 45% variant, 10% long-term holdout that stays on control for 30 days. It’s the only way to catch slow-burn issues like cache churn or origin connection pool exhaustion that won’t show up in a 72-hour test. And always check for Simpson’s Paradox: we’ve had tests where the variant won in every single PoP individually, but lost globally because the variant PoPs happened to serve more high-latency rural traffic that week. If your global numbers and your PoP-level numbers disagree, trust the segmented view.

Frequently Asked Questions