CDN Multi-Tenant Architecture: Isolating Customer Traffic While Sharing Infrastructure

A few years ago I got paged at 2 a.m. because one SaaS customer's flash sale had saturated three POPs and started serving another tenant's stale pricing JSON to everyone else. We were running what looked like a perfectly fine multi-tenant CDN — shared caches, shared connections, shared everything — until a single bad cache key collapsed the isolation we assumed we had. If you're building a SaaS CDN on shared infrastructure, that assumption will bite you. Here's how we now design for strict tenant isolation without giving up the economics of sharing.

Shared Infrastructure Without Shared Fate: What Actually Needs Isolating

When folks hear "multi-tenant CDN" they often picture separate configs per customer. That's not sustainable. At scale you're sharing the same edge servers, the same SSDs for cache, the same Anycast IPs, and the same TLS handshakes. The goal isn't to avoid sharing — it's to make sure sharing doesn't create fate-sharing. A burst, a misconfiguration, or a cache poisoning attempt from tenant A shouldn't degrade tenant B. I've seen teams try to solve this with brute force: dedicated POPs per enterprise customer. It works for two big logos, then your operational overhead triples and your cache hit ratio tanks because you've fragmented your footprint.

The isolation boundaries that matter most are cache, compute, and origin path. Cache is the obvious one. If two tenants serve /api/config.json but you don't include tenant context in your cache key, you're one collision away from a data leak. Compute is subtler: a regex-heavy WAF rule or image transformation for one tenant can starve CPU for everyone co-located on that box. And origin isolation is where SaaS platforms get hurt — if you route all tenants through a single origin connection pool without per-tenant circuit breakers, one slow origin will queue and timeout requests for neighbors. The spec fundamentals for how caching should behave here are still grounded in the HTTP Working Group Specifications, especially around Vary, cache keys, and stale-while-revalidate semantics.

We now treat tenant identity as a first-class input to every edge decision, not an afterthought you tack onto a URL path. That means extracting tenant ID at the earliest possible layer — from the Host, from a custom header, or from a verified token — and threading it through routing, caching, and logging. For a deeper look at the patterns we use, see CDN Tenant Isolation Patterns: Cache Separation, Rate Limits, and Access Controls. A typical Fastly-style VCL example looks like this:

sub vcl_hash {
    # Don't just hash req.url — include verified tenant context
    set req.hash += req.http.X-Tenant-ID;
    set req.hash += req.http.Host;
    set req.hash += req.url;
    # Isolate variants that matter for SaaS: auth, A/B, region
    if (req.http.Accept-Language) {
        set req.hash += req.http.Accept-Language;
    }
    return (lookup);
}

sub vcl_recv {
    # Extract tenant from vanity domain mapping table
    # table tenants { "customer1.example.com": "tnt_abc123", ... }
    set req.http.X-Tenant-ID = table.lookup(tenants, req.http.host);
    if (!req.http.X-Tenant-ID) {
        return (synth(421, "Unknown tenant"));
    }
    # Enforce per-tenant TLS requirement
    if (req.http.X-Tenant-Requires-mTLS && !client.cert.verified) {
        return (synth(403, "mTLS required"));
    }
}

Tenant-Aware Routing and Cache Keys on the Edge

Routing is where most shared infrastructure designs get sloppy. In a SaaS CDN you can't rely on path-based tenant IDs like /t/123/assets/app.js alone. Your customers want vanity domains — assets.customer.com CNAME'd to your edge — and that means tenant resolution has to happen via SNI and Host before you ever read the path. Back in 2019 we tried to do this mapping in our origin instead of at the edge, and every cache miss added 80-120ms while we looked up which tenant owned which domain. Moving that table to the edge, replicated via KV store with 5-second consistency, cut our p95 origin lookup time to under 4ms.

Once you've resolved tenant ID, your cache key has to be deterministic and non-colliding. We use a composite key of tenant_id + host + url + vary_headers and we hash it with SHA-256 at the edge. Don't forget to normalize — lowercasing hosts, sorting query params, stripping tracking params — or you'll fragment your cache and lose all the benefit of sharing. I also recommend namespace-prefixing your cache storage so you can purge by tenant without scanning the global keyspace. A purge for tnt_abc123 should only invalidate keys prefixed with that tenant, not trigger a global ban that degrades hit ratio for everyone else. For teams supporting white-labeled experiences, this is directly tied to White-Label CDN Delivery: Custom Branding for SaaS Customer Domains and Custom Domain CDN for SaaS: Automated SSL and DNS for Customer Vanity Domains, because automated certificates and CNAME validation are prerequisites for safe Host-based routing.

Here's a lightweight edge worker pattern we use today for tenant resolution, validation, and key augmentation before we hit cache. It's kept intentionally small to stay within CPU limits for per-request execution:

export default {
  async fetch(request, env, ctx) {
    const host = new URL(request.url).hostname.toLowerCase();
    // Tenant map replicated to edge KV — ~15k entries, <2ms lookup
    const tenantId = await env.TENANT_MAP.get(host, { type: "text" });
    
    if (!tenantId) {
      return new Response("Tenant not found", { status: 421 });
    }

    // Copy tenant context forward — never trust client-supplied header
    const headers = new Headers(request.headers);
    headers.set("x-tenant-id", tenantId);
    headers.set("x-cache-namespace", `tnt:${tenantId}`);

    // Build cache key with normalization
    const url = new URL(request.url);
    url.hostname = host;
    // Strip utm_* and sort remaining params for key stability
    const params = [...url.searchParams.entries()]
      .filter(([k]) => !k.startsWith("utm_"))
      .sort(([a],[b]) => a.localeCompare(b));
    url.search = new URLSearchParams(params).toString();

    const cacheKey = new Request(url.toString(), {
      headers,
      method: request.method
    });

    // Add tenant-scoped rate limit check (token bucket in KV/Durable Object)
    const allowed = await checkTenantRateLimit(tenantId, env);
    if (!allowed) {
      return new Response("Too Many Requests", { status: 429, headers: {
        "Retry-After": "2",
        "x-tenant-id": tenantId
      }});
    }

    return fetch(cacheKey, { cf: { cacheTtl: 3600, cacheNamespace: tenantId } });
  }
}

async function checkTenantRateLimit(tenantId, env) {
  // 10k req/s burst, 5k sustained — counters in Durable Object
  const obj = env.RATE_LIMITER.get(env.RATE_LIMITER.idFromName(tenantId));
  const res = await obj.fetch("https://limiter/check");
  return res.status === 200;
}

The Noisy Neighbor Problem: Quotas, Contention, and Fair Scheduling

Isolation doesn't mean much if one tenant can hog all the resources on a shared POP. I've watched a single customer running unoptimized image resizing — 8K source files, no width limits — drive edge CPU to 92% sustained and push p99 latency for every other tenant on that box from 45ms to almost 400ms. You can't fix that with autoscaling alone; you need fair scheduling at the edge. That means per-tenant concurrency caps, CPU time quotas, and bandwidth throttles enforced before requests ever queue for origin.

How you enforce quotas depends on where you terminate. If you're on NGINX or Envoy at the edge, use separate filter chains per tenant class (free vs. pro vs. enterprise) with token bucket rate limiters and connection limits. On serverless edge runtimes, isolate with per-tenant Durable Objects or per-tenant queues. And please, set timeouts and circuit breakers per origin per tenant. We give each tenant origin pool its own circuit breaker with 50% error threshold over 30 seconds, plus a bulkhead of 100 concurrent connections. When that trips, only that tenant gets 503s with Retry-After; everyone else stays healthy. Modern guidance on performance budgets and response handling from web.dev Articles is useful here for setting realistic SLOs around INP and LCP under contention.

Putting Tenant Isolation Into Practice: Edge Config That Actually Works

We've talked theory — now let's make it real. In production, tenant isolation lives in three places: your edge logic, your cache key schema, and your origin routing. Miss one and you've got a leaky abstraction that'll bite you at 2 AM when Tenant A's purge wipes Tenant B's assets. The pattern we use for most SaaS platforms serving 5,000-50,000 tenants off a shared fleet is deceptively simple: extract tenant context as early as possible — ideally at TLS termination — and carry it as an immutable header through the entire request lifecycle.

Don't rely on URL path parsing alone. Custom domains, wildcard subdomains, and vanity domains will break you. Instead, map every incoming hostname + SNI to a tenant_id via an edge dictionary (we keep ours in Workers KV / DynamoDB Global Tables with < 5ms lookup p95). That lookup also returns the tenant's tier, rate limit bucket, and cache policy. If you can't resolve a tenant in under 10ms, fail open to a 421 Misdirected Request — don't guess. We've seen platforms try to infer tenant from a JWT after caching; that's how you end up serving one customer's private invoice to another.

Once you have that tenant_id, everything else becomes deterministic. Your cache key must include it, your logs must include it, and your CDN Usage Tracking for SaaS: Per-Customer Bandwidth and Request Analytics pipeline should aggregate on it at the edge rather than trying to reconstruct it from access logs 24 hours later. For billing-grade accuracy, you'll want to emit usage counters directly from the edge — check out how we approach API Metering at CDN Edge: Real-Time Usage Tracking for SaaS Billing to avoid the classic 15-minute log delay that breaks bursty API customers. Here's a stripped-down Cloudflare Worker example we use as a template:

// Tenant-aware edge router - runs on every PoP, <1ms overhead
export default {
  async fetch(request, env) {
    const host = new URL(request.url).hostname;
    
    // 1. Resolve tenant in <5ms from edge dictionary (KV / Edge Config)
    const tenant = await env.TENANT_MAP.get(host, { type: "json" });
    if (!tenant) return new Response("Unknown tenant", { status: 421 });

    // 2. Enforce per-tenant rate limit (token bucket via Durable Object)
    const limiterId = env.RATE_LIMITER.idFromName(tenant.id);
    const limiter = env.RATE_LIMITER.get(limiterId);
    if (!(await limiter.check(tenant.tier_rps))) {
      return new Response("Too Many Requests", { status: 429, headers: {
        "Retry-After": "1", "X-Tenant-ID": tenant.id
      }});
    }

    // 3. Build tenant-isolated cache key - NEVER omit tenant.id
    const cacheKey = `${tenant.id}::${request.url}::${request.headers.get("Accept-Encoding") || "identity"}`;
    let response = await caches.default.match(cacheKey);
    if (response) {
      response = new Response(response.body, response);
      response.headers.set("X-Cache", "HIT");
      response.headers.set("X-Tenant-ID", tenant.id);
      return response;
    }

    // 4. Route to tenant's origin shard with isolated headers
    request.headers.set("X-Tenant-ID", tenant.id);
    request.headers.set("X-Origin-Shard", tenant.shard);
    let originResp = await fetch(request);
    
    // 5. Emit real-time usage (fire-and-forget, sampled 100% for billable tenants)
    env.ANALYTICS.writeDataPoint({
      blobs: [tenant.id, host],
      doubles: [originResp.headers.get("Content-Length") || 0],
      indexes: [tenant.id]
    });

    originResp = new Response(originResp.body, originResp);
    originResp.headers.set("X-Cache", "MISS");
    // Isolate cache write per-tenant with tier-aware TTL
    env.waitUntil(caches.default.put(cacheKey, originResp.clone()));
    return originResp;
  }
}

Advanced Patterns and Troubleshooting When Isolation Breaks

Once you're past 10k tenants, simple header-based isolation isn't enough. You'll hit control-plane limits — think 100k cache purge rules, 500+ custom certificates per PoP, or noisy neighbor contention on shared SSD cache. That's where we move to cell-based architecture. Instead of one global cache pool, we shard tenants into cells of ~500-1,000 tenants per isolated cache and compute partition. Each cell gets its own rate limiter, purge queue, and even its own Anycast IP if you're paranoid. It adds about 8-12% infra overhead, but it means a cache stampede from one cell can't evict hot objects in another. We automate cell assignment based on traffic — enterprise tenants with >10k RPS or >500 Mbps sustained get dedicated cells; everyone else shares.

The trickiest leaks aren't dramatic — they're subtle. The top three we debug: first, cache key collisions from normalized URLs (e.g., stripping query strings that actually contain tenant context like ?org_id=123). Second, origin header bleed where your origin returns Vary: Cookie and the edge accidentally coalesces responses across tenants. And third, log enrichment failures where your SIEM shows 20% of requests with tenant_id=unknown because health checks and bot traffic bypass your resolver. Our fix is brutal but effective: we run a synthetic canary every 30 seconds per PoP that requests tenant-a.example.com/tenant-check and asserts it never gets a cache HIT with tenant-b's ETag. If it does, we auto-drain that PoP.

Don't forget identity either. When enterprise customers bring their own IdP, your edge auth logic becomes part of the isolation boundary. You can't just validate a JWT and call it a day — you need to bind the token's iss and aud to the resolved tenant_id, otherwise a valid token from Tenant A could be replayed against Tenant B's custom domain. We've standardized on Enterprise SSO Integration with CDN: SAML, OIDC, and Custom Identity Providers patterns where the edge does that binding before it even checks the cache. If you see 403s spiking for a single tenant after an IdP rotation, check your JWKS cache TTL — we've seen it default to 6 hours and cause 45-minute auth outages.

Frequently Asked Questions