Private CDN Deployment: Building Enterprise-Owned Edge Infrastructure

Back in 2019 I got a call at 2 AM because a public CDN's global purge queue had stalled and our largest SaaS customer couldn't clear stale pricing data. We'd done everything right on our end, but we had zero visibility into their control plane and no way to force the invalidation. That's the moment our team stopped treating CDN as a rented service and started planning our own edge. If you're serving regulated data, custom logic at the edge, or just pushing enough terabytes that the bandwidth bill hurts, that same conversation is probably coming for you.

Why Enterprises Are Reclaiming Control at the Edge

Most teams don't start by wanting to run their own CDN. They start with a public provider and hit a wall. For us it was compliance. We had a healthcare workload that couldn't let cache keys or logs leave a specific geography, and no amount of contract addendums could guarantee that edge nodes outside that boundary wouldn't temporarily hold data. For other orgs I've advised, it's been the need to run proprietary WAF rules, image optimization that depends on internal ML models, or request collapsing logic that public vendors won't support. When your product's logic has to live within 20ms of the user, renting generic compute at the edge stops being enough.

There's also a cost inflection point that no one likes to talk about. Public CDN pricing is fantastic until you cross 300-500 TB/month of egress with a high cache hit ratio. At that volume you've already optimized origins, and you're essentially paying a premium for bandwidth you could buy directly from transit providers. I've seen teams cut their delivery cost by 40-60% after moving to owned infrastructure, but only after they accounted for the operational overhead. You aren't just buying servers; you're buying the responsibility for BGP, capacity planning, and cache invalidation consistency. If you don't staff for that, you haven't built a CDN Multi-Tenant Architecture: Isolating Customer Traffic While Sharing Infrastructure — you've built a very expensive reverse proxy farm.

That doesn't mean you need to replicate Cloudflare's 300+ PoPs on day one. A practical private deployment often starts with 4-8 strategic PoPs where your users actually are. We've had success placing enterprise edge nodes in Equinix metros that peer directly with the ISPs our customers use, rather than chasing geographic coverage for its own sake. The HTTP caching semantics you rely on don't change — we still implement validation and freshness per HTTP Working Group Specifications (RFC 9111) — but you finally own where that logic executes. And for guidance on client-side performance implications, the patterns documented in web.dev Articles on caching and delivery remain invaluable even when the edge is yours.

# /etc/nginx/nginx.conf - minimal private edge node with origin shielding
proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=EDGE:256m 
                 inactive=24h max_size=500g use_temp_path=off;

upstream origin_shield {
    # Shield PoP - only this tier talks to origin
    server shield.internal.example.com:443 max_fails=3 fail_timeout=30s;
}

server {
    listen 443 ssl http2;
    ssl_certificate /etc/ssl/edge.example.com.crt;
    ssl_certificate_key /etc/ssl/edge.example.com.key;

    location / {
        proxy_cache EDGE;
        proxy_cache_key $scheme$host$request_uri$http_accept;
        proxy_cache_valid 200 302 1h;
        proxy_cache_valid 404 1m;
        proxy_cache_lock on;
        proxy_cache_use_stale error timeout updating http_500 http_502;
        proxy_cache_background_update on;
        
        # Serve stale while revalidating per RFC 9111
        proxy_cache_revalidate on;
        add_header X-Cache-Status $upstream_cache_status;
        
        proxy_pass https://origin_shield;
        proxy_next_upstream error timeout http_500;
    }
    
    # Instant purge endpoint (authenticated internally)
    location ~ /purge(/.*) {
        allow 10.0.0.0/8;
        deny all;
        proxy_cache_purge EDGE "$scheme$host$1$http_accept";
    }
}

Architecting the Private PoP: Hardware, Software, and Anycast

Don't overbuild the server. For a cache-heavy enterprise workload, I've found a 2U box with dual 32-core CPUs, 256GB RAM, and 8x 7.68TB NVMe drives in RAID0 for cache will outperform a fancier setup with smaller disks and more RAM than you can use. Your bottleneck is almost always NVMe throughput and NIC capacity, not CPU, unless you're doing heavy TLS termination or edge compute. We standardized on 100G NICs with DPDK-capable drivers even when we only bought 20G of transit — it gave us headroom for traffic spikes without re-racking. For software, Apache Traffic Server and Varnish still dominate private deployments because they give you explicit control over cache storage and eviction; Nginx is simpler if your private CDN is mostly about shielding and routing rather than complex VCL.

The harder part is the network fabric. A private CDN without Anycast isn't really a CDN — it's just a set of distributed origins. We run BGP Anycast from each PoP using the same /24 announced from multiple metros, paired with a health-checked withdraw. If a PoP's cache tier fails health checks, it automatically withdraws its announcement and traffic reconverges to the next closest site in under 60 seconds. You'll need your own ASN and at least a /24 of IPv4 space; you can't effectively Anycast anything smaller and expect global propagation. Don't try to build this on top of your corporate WAN — get dedicated transit and peering at each site. Your users' ISPs will thank you with lower latency.

Operational isolation matters as much as routing. Even when the infrastructure is owned, your SaaS customers are still sharing those edge nodes, so you need strict tenant boundaries at the cache and compute layer. We learned this after one customer's aggressive cache-busting query strings evicted another customer's hot objects. Now every tenant gets a segmented cache namespace, dedicated rate limit counters, and per-tenant access logs. That pattern is well described in CDN Tenant Isolation Patterns: Cache Separation, Rate Limits, and Access Controls and it's non-negotiable for private deployments. The same goes for branding — if you're delivering on behalf of enterprise clients, you'll want White-Label CDN Delivery: Custom Branding for SaaS Customer Domains with per-tenant TLS certs and CNAME handling automated at provisioning time.

# /etc/bird/bird.conf - Anycast announcement for private CDN PoP (ams-01)
router id 203.0.113.10;

protocol device { scan time 10; }

protocol kernel {
    ipv4 { export all; }
}

protocol bgp transit1 {
    local 203.0.113.10 as 64512;
    neighbor 203.0.113.1 as 64500;
    ipv4 {
        import none;
        export filter {
            if (cdn_pop_healthy() = 1) then {
                # Announce owned Anycast /24 only when healthy
                if net = 198.51.100.0/24 then accept;
            }
            reject;
        };
    }
}

# Health check script toggles file presence
function cdn_pop_healthy() {
    if (roa_check("/var/run/cdn_healthy", 1)) return 1;
    return 0;
}

Sizing, Budgeting, and What Owned Infrastructure Really Costs

Capacity planning for a private deployment is where many teams get burned because they model for average traffic instead of thundering herds. We size each PoP to handle 2.5x our p95 peak for that region, and we test eviction behavior under steady-state fill. If your working set is 2TB of hot objects and you've only provisioned 1TB of NVMe cache, your hit ratio will crater regardless of how good your caching headers are. I always run a week-long dark launch where the private PoPs mirror live traffic but aren't in the Anycast path yet — you get real hit ratio data without risking users.

Budgeting isn't just capex versus opex. The table below is based on numbers from a recent 400 TB/month SaaS deployment across North America and EU — four owned PoPs versus a comparable enterprise contract with a top-tier public CDN. The private option looked cheaper on bandwidth alone, but once we added staff (two network engineers, one SRE dedicated to edge), transit, colocation, and hardware amortization over 36 months, the breakeven stretched to month 14. You don't save money in quarter one. You save money in year two, and you gain control from day one.

What you're really buying is determinism. With owned infrastructure, purge propagation is measured in milliseconds within a PoP and under two seconds globally because you control the message bus. Cache hit ratio is yours to tune without abstraction layers, and compliance audits become trivial when you can show auditors the physical cage, disk encryption status, and log pipeline. For SaaS providers handling enterprise contracts with data residency clauses, that auditability often justifies the build even if the bandwidth economics are neutral.

Implementation Blueprint: From Bare Metal to Serving Traffic

Alright, you've got the architecture diagram and the budget approval — now it's time to actually rack it. We've found the fastest path to a live PoP isn't building a perfect, snowflake-optimized server. It's standardization. Pick one hardware SKU and clone it everywhere. Our current reference is a 2U box: dual AMD EPYC 7313s, 256GB RAM, 2x 3.84TB NVMe for hot cache, and 6x 15TB SSDs for warm tier, with dual 25GbE NICs in LACP. You’ll PXE-boot Ubuntu 22.04 LTS, automate with Ansible, and have the whole site provisioned in under 90 minutes. Don't overthink the OS tuning at first — set your NIC ring buffers to 4096, enable RPS/XPS, and bump your file descriptor limits to at least 1M. You'll tune the rest once you have real traffic.

For software, don't try to build your own cache — pick a proven proxy and put your energy into automation. We run ATS (Apache Traffic Server) for pure throughput and Varnish where we need complex VCL logic, both fronted by Nginx for TLS termination and HTTP/3. The real work is automating certificate and origin management. If you're serving SaaS customers on their own domains, manual cert uploads will kill you at 50+ domains. You need ACME automation with DNS-01 challenges from day one. We've automated issuance for thousands of vanity domains using Let's Encrypt with a 7-day renewal buffer, and it’s the only way to keep your 99.99% SSL success rate. For a deeper dive on this, see our guide on Custom Domain CDN for SaaS: Automated SSL and DNS for Customer Vanity Domains.

The last mile is announcing your space and closing the loop on observability. You'll peer via BGP using BIRD2 — announce your /24 (never smaller if you want global propagation) from at least two transits per PoP to avoid single-homed blackholes. Set your BGP timers to 30s hold-time and use BFD for sub-second failure detection; we've cut failover from 90 seconds to under 3 seconds this way. Before you push more than 10% of production traffic, validate with synthetic checks from Catchpoint or ThousandEyes. Your cache hit ratio should climb to 92-96% within 24 hours if your TTLs are sane. If it's stuck at 70%, you’ve got a cache-key problem, not a hardware problem.

# /etc/bird/bird.conf - Minimal Anycast PoP Announcement
# + Varnish VCL snippet for sane cache keys
protocol bgp upstream1 {
  local as 65001;
  neighbor 203.0.113.1 as 174;
  import filter { accept; };
  export filter { if net ~ 203.0.113.0/24 then accept; else reject; };
  bfd on;
}

# Varnish VCL - Normalize cache key and handle shield
sub vcl_hash {
  hash_data(req.url);
  # Don't cache-bust on trivial headers
  if (req.http.host) {
    hash_data(req.http.host);
  }
  return (lookup);
}

sub vcl_backend_response {
  # 1h TTL for static, 2s grace + 10s keep for thundering herd protection
  if (beresp.http.Content-Type ~ "image|font|text/css|javascript") {
    set beresp.ttl = 1h;
    set beresp.grace = 10s;
  }
  # Enable stale-while-revalidate
  set beresp.http.Stale-While-Revalidate = "30";
}

Advanced Edge Patterns and Troubleshooting at Scale

Once you've got three or more PoPs up, a flat architecture starts to hurt. You’ll see origin egress bills spike because every PoP is pulling the same 4K hero image independently. The fix is tiered caching with an Origin Shield. Designate one PoP per region (we use Frankfurt for EMEA, Ashburn for NA, Singapore for APAC) as the shield — all other edge PoPs fetch from the shield, not the origin. That single change cut our origin fetch volume by 68% and dropped origin p95 latency from 340ms to 85ms for cache misses. Pair it with request coalescing (a.k.a. collapsed forwarding) so when 2,000 clients request the same uncached asset at once, only one request goes upstream. Without it, you’ll Thundering Herd your own origin on every purge.

You'll also want to push logic to the edge that used to live in your core. This is where private infrastructure pays for itself. Need per-customer rate limiting or usage-based billing? Don't hairpin back to an API gateway. We run Lua in Nginx/OpenResty to count requests and bandwidth at the edge and stream counters to ClickHouse via Vector every 10 seconds. That gives us real-time metering with <1% drift vs. origin logs and lets us enforce 10k RPS per API key without adding a single millisecond. If you're monetizing the edge, check out API Metering at CDN Edge: Real-Time Usage Tracking for SaaS Billing and our breakdown of CDN Usage Tracking for SaaS: Per-Customer Bandwidth and Request Analytics — doing this correctly is the difference between knowing your top 10 customers and guessing.

When things break — and they will — check these three first. First, if your hit ratio suddenly drops 15 points, you've likely got a cache-key explosion. Look for a new `Vary: Cookie` or a random query param like `?v=timestamp` that someone added upstream. Fix it by normalizing the key in VCL and stripping marketing params (`utm_*`, `fbclid`). Second, intermittent 502s at high concurrency almost always mean ephemeral port exhaustion or `net.ipv4.ip_local_port_range` too small — bump it to 1024-65535 and enable `tcp_tw_reuse`. We hit this at ~18k RPS per box. Third, if Anycast is sending Berlin users to New York, your BGP community strings are wrong or you've got a stuck route at a peer. Withdraw the prefix, check your looking glass, and remember: Anycast is not GeoDNS. You control proximity with BGP, not DNS tricks.

Frequently Asked Questions