Traffic Splitting for Progressive Rollouts: Weighted Routing at the CDN Layer

Progressive rollouts live or die on how precisely you can control traffic. In my experience running edge deployments serving 50M+ requests per day, the difference between a smooth 1% canary and a 2 AM incident is rarely the application code itself — it's the routing layer that decides who sees what and how quickly you can reverse it. DNS-based weighting has its place, but when you need sub-minute shifts, per-request consistency, and instant rollback without cache invalidation storms, weighted routing at the CDN layer is the control plane you want. This article breaks down how to implement reliable traffic splitting at the edge, what to watch forOperationally, and how to avoid the subtle failure modes that catch teams off guard.

Why CDN-Layer Splitting Outperforms DNS-Based Weighted Routing for Rollouts

For years, teams defaulted to DNS for traffic shifting because it was simple: create two weighted A records, 95% to the stable pool and 5% to the canary, and let resolvers do the work. The problem is DNS was never designed for rapid, deterministic experiment control. In practice, I've seen three persistent issues that make DNS weighting unsuitable for progressive rollouts beyond very coarse, long-lived splits.

First is cache persistence. Even with a 30-second TTL, intermediate resolvers, ISP caches, and client OS caches routinely hold records for minutes to hours. When you need to move from 5% to 0% because error rates spiked, you are not recalling traffic — you are waiting for caches to expire. At the CDN edge, a configuration or worker deployment propagates globally in 5-15 seconds and applies to the very next request.

Second is lack of request-level stickiness. DNS routing is per-hostname lookup, not per-user. A single user refreshing a page can flip between origins if their resolver re-queries or round-robins, which pollutes your experiment data and creates a jarring experience. Edge routing can pin a user cohort deterministically.

Third is blindness to HTTP context. DNS doesn't know if the request is for an API endpoint you want to canary, a static asset you don't, or a bot you should exclude. The edge does.

When DNS Weighting Still Makes Sense

This doesn't mean DNS is obsolete. For multi-region failover where you need to shift entire PoPs or data centers — for example, draining a region for maintenance — DNS and Anycast DNS Architecture: How CDNs Route Users to the Nearest Edge remain the right tool. The distinction is scope: DNS controls which edge cluster handles the request, while edge weighted routing controls which origin or code path inside that cluster does. I've found that the most resilient global architectures use both in layers. DNS gets the user to the nearest healthy edge, and the edge worker makes the fine-grained rollout decision. If you are coupling this with Global Server Load Balancing: GSLB Strategies for Multi-Region Deployments, treat GSLB as the coarse dial and edge traffic splitting as the fine dial.

The Latency Advantage of Edge Decisions

Another underappreciated benefit is latency. A DNS-based split that points 10% of users to a far-away canary region adds 80-120ms of round-trip time for those users. An edge-based split keeps the TLS termination and routing logic at the nearby PoP — typically under 20ms from the user — and then proxies to the correct origin over the CDN's optimized backbone. The user never pays a DNS distance penalty for being in an experiment.

// Example: Naive DNS-weighted approach (limited control)
// DNS Zone - cannot target by URI, header, or cookie
// api.example.com.  30  IN  A  203.0.113.10 ; weight 90 - stable
// api.example.com.  30  IN  A  203.0.113.20 ; weight 10 - canary
// Problem: Resolver caching makes rollback slow, no stickiness

// Edge-weighted approach (precise, instant)
// Executed on every request at the edge PoP - see next section
// for full implementation. Rollback is a config push, not TTL expiry.

Weighted Routing Mechanics: Consistent Hashing, Cookie Affinity, and Request Sampling

The core of any progressive rollout system is how you decide that a given request belongs to bucket A or bucket B. Random percentage checks with Math.random() < 0.05 will technically give you 5% traffic, but they break session continuity and make debugging miserable. In my experience, deterministic hashing is the only production-viable approach for anything user-facing.

The pattern is straightforward: take a stable identifier for the user or request, hash it, normalize to 0-99, and compare against your weight threshold. That identifier might be a user ID, an anonymous device ID, or for unauthenticated traffic, the combination of IP + User-Agent. The key property is that the same user consistently maps to the same bucket until you change the weight.

For authenticated traffic, hash the user ID. For anonymous traffic, I recommend setting a lightweight edge cookie cdn_bucket_id with a UUID on first request and hashing that. Hashing on IP alone fails under CGNAT where thousands of mobile users share an IP, and it violates privacy best practices in some regions.

Implementing Deterministic Splitting in an Edge Worker

Below is a pattern I've used on Cloudflare Workers, Fastly Compute@Edge, and AWS CloudFront Functions. The logic is portable — the APIs change, but the hashing and header propagation remain the same.

export default {
  async fetch(request, env, ctx) {
    const url = new URL(request.url);
    
    // Only split API traffic - let static assets stay on stable
    if (!url.pathname.startsWith('/api/')) {
      return fetch(request);
    }

    // 1. Resolve stable bucket identifier
    let bucketId = request.headers.get('x-user-id');
    if (!bucketId) {
      const cookie = request.headers.get('Cookie') || '';
      const match = cookie.match(/cdn_bucket_id=([^;]+)/);
      bucketId = match ? match[1] : null;
    }

    // 2. Deterministic hash to 0-99 (FNV-1a is fast and sufficient)
    function hashToBucket(str) {
      let hash = 2166136261;
      for (let i = 0; i < str.length; i++) {
        hash ^= str.charCodeAt(i);
        hash = Math.imul(hash, 16777619);
      }
      return Math.abs(hash) % 100;
    }

    const CANARY_WEIGHT = 5; // 5% rollout - promote via config
    let isCanary = false;
    
    if (bucketId) {
      isCanary = hashToBucket(bucketId) < CANARY_WEIGHT;
    } else {
      // First-time anonymous user - sample and set affinity cookie
      isCanary = Math.random() * 100 < CANARY_WEIGHT;
    }

    // 3. Route and propagate decision for origin + observability
    const originUrl = isCanary 
      ? 'https://canary-origin.example.com' 
      : 'https://stable-origin.example.com';
    
    let newRequest = new Request(url.toString(), request);
    newRequest.headers.set('x-cdn-split', isCanary ? 'canary' : 'stable');
    newRequest.headers.set('x-cdn-bucket', bucketId || 'new');

    let response = await fetch(newRequest, { backend: isCanary ? 'canary' : 'stable' });

    // 4. Set affinity cookie for anonymous users if not present
    if (!bucketId) {
      const newId = crypto.randomUUID();
      response = new Response(response.body, response);
      response.headers.append('Set-Cookie', `cdn_bucket_id=${newId}; Path=/; Max-Age=86400; SameSite=Lax`);
    }
    
    response.headers.set('x-cdn-split-served', isCanary ? 'canary' : 'stable');
    return response;
  }
}

A few details from production that matter: Always emit the routing decision as a request header (x-cdn-split) and a response header. Your origins need to log it for correct analysis, and your RUM pipeline needs it to segment performance. Also, keep your hashing algorithm stable. Changing from FNV to Murmur mid-rollout will reshuffle all users and invalidate your before/after data. As defined in the HTTP Working Group Specifications, preserving header semantics and avoiding unintended caching of variant responses is critical — add Vary: Cookie, x-cdn-split where appropriate or, better, make the split header internal-only and strip it before caching.

Architecting Sticky Versus Stateless Splits for User Cohort Integrity

Once you have hashing, you need to decide what consistency model you promise. I classify splits into two categories, and choosing the wrong one is a common source of rollout noise.

Sticky splits guarantee that once a user is assigned to canary, they stay on canary for the duration of the rollout phase. This is achieved with a cookie or a deterministic hash as shown above. Sticky is mandatory when the canary involves stateful changes: new API contracts, database schema writes, or UI mutations where a user flipping mid-session would see errors or data inconsistency. The cost is complexity — you need cookie handling, cache key variation, and GDPR-aware cookie consent if in the EU.

Stateless splits make a per-request decision without affinity, often using random sampling. This is acceptable for purely backend, idempotent changes like a new image optimization pipeline or a read-only service refactor where request-to-request consistency doesn't affect correctness. Stateless is simpler and avoids cookie overhead, but it doubles your variance — the same user may contribute latency samples to both buckets.

In my experience, the safest default is sticky for any user-facing progressive delivery. The debugging payoff is enormous. When a user reports a bug, you can ask for their x-cdn-split-served value or check the affinity cookie and instantly know which code path they hit.

Cache Partitioning and Key Hygiene

The most subtle failure I see is cache pollution. If your CDN cache key doesn't include the split decision, a canary response can be cached and served to stable users, or vice versa. For Varnish/Fastly VCL or CloudFront cache policies, you must either:

1. Bypass cache for split traffic entirely during the canary window (safest for low-traffic APIs), or

2. Add the split identifier to the cache key: cache_key = hash(url) + split_bucket.

Option 2 preserves hit ratio while maintaining isolation. One team I worked with kept a 92% hit ratio during a 10% canary by keying on x-cdn-split — versus dropping to 45% when they bypassed. For HTML documents, also consider interaction with DNS Prefetching and Preconnect: Reducing Connection Setup Latency. If your canary origin lives on a different domain (canary-origin.example.com), a preconnect hint to the stable origin won't help canary users. Keep origins behind the same hostname and switch internally at the edge to avoid that penalty.

Automated Guardrails: Health Checks, Error Budgets, and Instant Rollback at the Edge

Manual weight increases are risky. You will forget to check an error rate, or you will be asleep when the canary starts 500-ing. Every weighted routing system I ship now includes three guardrails: origin health awareness, error-budget-based auto-rollback, and a single-click kill switch.

Health-aware shifting means the edge load balancer continuously probes both origin groups. If canary health drops below a threshold (e.g., two consecutive probe failures or p75 origin latency > 800ms), the edge should automatically drain traffic from that group regardless of configured weight. This is distinct from traditional CDN Failover and Origin Redundancy: Ensuring Availability During Outages — failover is binary, while health-aware splitting is proportional. You still honor the weight when healthy, but you degrade gracefully when not.

Auto-rollback watches golden signals: 5xx rate, p95 latency, and origin error ratio segmented by split bucket. I typically set a relative threshold: if canary 5xx rate exceeds stable by more than 0.5% absolute or 200% relative for two minutes, automatically set effective weight to 0% and page the rollout owner. The weight configuration remains 5% in the control plane — the runtime override is what drops.

Implementation-wise, avoid coupling rollback logic to a centralized controller that polls. Push the thresholds into the edge config and let each PoP evaluate locally. That way rollback is not dependent on a control-plane API being reachable during an incident.

# Example: Edge origin group with health-aware weighted routing
# This is Fastly-style VCL / Terraform-inspired config - adapt to your CDN
resource "edge_origin_group" "api" {
  name = "api-rollout"

  origin {
    name   = "stable"
    host   = "stable-origin.example.com"
    weight = 95
    healthcheck {
      path     = "/healthz"
      interval = 5
      timeout  = 2
      threshold_healthy = 3
    }
  }

  origin {
    name   = "canary"
    host   = "canary-origin.example.com"
    weight = 5
    healthcheck {
      path     = "/healthz"
      interval = 5
      timeout  = 2
      threshold_healthy = 2
    }
  }

  # Guardrail: automatic drain
  auto_drain {
    enabled             = true
    error_rate_window   = "2m"
    error_rate_threshold = 0.015  # 1.5% 5xx in canary vs 0.3% in stable
    latency_p95_threshold_ms = 850
    action              = "set_effective_weight_zero"
    notify_channel      = "pagerduty-rollouts"
  }

  # Sticky affinity via cookie
  affinity {
    type       = "cookie"
    cookie_name = "cdn_bucket_id"
    ttl_seconds = 86400
  }
}

Keep your kill switch mechanical: a single feature flag or config value that sets canary weight to zero and bypasses all hashing. In an incident, you don't want to reason about hash ranges — you want a button that says “100% stable now.” Test it monthly with a synthetic 10% canary to a black-hole origin and verify traffic drops to zero in under 15 seconds.

Comparing Edge Execution Models: Worker Logic Versus Origin Load Balancer Versus DNS

Teams often ask where to implement the split. The honest answer is it depends on rollout velocity and required granularity, but the trade-offs are consistent across providers I've evaluated.

I've found that origin load balancer weights are too blunt for most product rollouts. They can't say “canary only for /api/v2/orders when header x-experiment: checkout-rewrite is present,” which is exactly the precision product teams need. Workers give you that expressiveness at the cost of owning a small codebase. My rule of thumb: if you roll out more than twice a month or need stickiness, invest in the worker. If you shift traffic quarterly for capacity reasons, the load balancer is sufficient.

Cost is also a factor people underestimate. Workers incur per-request compute charges — typically $0.15-$0.60 per million requests. For a 10k RPS API, a worker-based split adds roughly $400-$900/month. DNS weights have no per-request cost but hide the cost of cache misses and rollback latency. In every post-incident review I've participated in where DNS weighting delayed rollback, the incident cost dwarfed a year of worker fees.

Measuring Rollout Success: Edge Telemetry, Synthetic Probing, and Real User Metrics

A traffic split without split-aware observability is just load distribution. To validate a progressive rollout, you need to prove three things: exposure was correct, errors didn't regress, and performance didn't degrade for the canary cohort.

Start with exposure logging. Every request that passes through the edge must log split_bucket, weight_at_time, and bucket_id_hash (or a truncated version for privacy). Without weight-at-time, you cannot reconstruct whether a 5% weight actually delivered 5% exposure — clock skew or config drift can cause divergence. Ship these fields to your logging pipeline as structured JSON, not as free-form access log strings.

Segmented Telemetry Pipeline

On the metrics side, don't average across buckets. I create separate dashboards for canary vs. stable that mirror each other: request rate, 5xx ratio, origin latency p50/p95/p99, cache hit ratio, and edge compute duration. The relative deltas matter more than absolutes. A canary that increases p95 by 15ms may be acceptable if it carries a new feature; a 0.2% error rate increase is not.

Real User Monitoring needs the same segmentation. Inject the served bucket as a custom dimension in your RUM beacon. That lets you compare Core Web Vitals for stable vs. canary on actual devices. I've caught cases where origin latency looked flat but LCP regressed by 250ms on Android because the canary payload included an unoptimized hero image — something origin metrics never showed. For context on related header and protocol behaviors, the IETF Datatracker extensible prioritization specifications are a useful reference when you reason about how canary responses are prioritized over HTTP/2 and HTTP/3.

Synthetic Probes and Gradual Promotion Checklist

Before promoting weight, run synthetic probes pinned to each bucket. A simple cron that sends requests with Cookie: cdn_bucket_id=forced-canary-test-id mapped to a known canary hash lets you continuously validate both origins independent of live traffic mix. I require these probes to pass for 10 minutes before any weight increase.

My promotion sequence that has survived multiple high-risk migrations is: 1% for 30 minutes, 5% for 2 hours, 10% for 6 hours, 25% for 12 hours, 50% for 12 hours, then 100%. Each step requires that error delta, latency delta, and business KPIs (conversion, add-to-cart) stay within the predefined guardrail. If you need to move faster, compress the time but never skip the error-budget check. The edge weight change itself should be a single atomic config publish — not a rolling origin deploy — so you can promote without redeploying code.

Frequently Asked Questions