CDN Overlay Networks: VXLAN, GRE, and WireGuard Tunneling at Scale

Last winter we had a PoP in Frankfurt that couldn't advertise new cache prefixes without flapping the entire underlay. The BGP session was healthy, the fabric was healthy, yet every time we pushed a new service subnet we risked reconvergence across 40+ routers. That's the moment most CDN teams realize the physical network can't be your service boundary. Overlay networks let us decouple service routing from the underlay, and at our scale — 250+ PoPs, 80 Tbps peak — you don't get a choice between VXLAN, GRE, and WireGuard. You'll run all three, but you'd better know where each one hurts.

The Underlay Isn't Your Service Network: Decoupling Fabric From Function

When I started building CDN fabrics ten years ago, we tried to do everything in the underlay. One big IS-IS domain, BGP everywhere, VLANs stretched cautiously where we absolutely needed them. It worked until we hit about 50 PoPs. After that, operational complexity exploded. Adding a new tenant or an isolated midgress tier meant touching core configs, waiting for maintenance windows, and praying you didn't create a transient loop. We'd spend more time coordinating than configuring.

Overlays fixed that by giving us a separate control plane for services. The underlay's only job now is to provide resilient, ECMP-capable IP reachability between VTEPs and tunnel endpoints. We run a simple eBGP fabric with ECMP Routing in CDN Networks: Equal-Cost Multi-Path Traffic Distribution doing the heavy lifting for path diversity — no cleverness there, just fast reconvergence and massive multipath. Service state — VNIs, VRFs, tunnel keys — lives entirely in the overlay. If an overlay endpoint dies, the underlay doesn't even notice. If the underlay loses a spine, BFD pulls the path in 300ms and the tunnels rehash without dropping customer flows. That separation is what lets us deploy new cache clusters in hours instead of weeks.

Don't confuse this with MPLS in CDN Backbone Networks: Label Switching for Predictable Performance or Segment Routing for CDN Traffic Engineering: Source-Routed Path Selection. We still use MPLS and segment routing in our long-haul backbone to engineer inter-PoP paths with predictable latency. The overlay sits *on top* of whatever transport you choose. I've seen teams try to use SR policies as a substitute for overlays and end up with a nightmare of per-service SID lists. Underlay gives you reachability and traffic engineering; overlay gives you isolation and agility. Keep those jobs distinct and you'll sleep better.

# VXLAN VTEP on Cumulus / FRR - stripped down production config
# Underlay: eBGP unnumbered, Overlay: EVPN for VXLAN control plane
auto lo
iface lo inet loopback
    address 10.255.0.11/32  # Router-ID / VTEP source

auto vxlan10101
iface vxlan10101
    vxlan-id 10101
    vxlan-local-tunnelip 10.255.0.11
    bridge-access 10101       # Maps to VNI 10101 -> tenant: image-optimization
    mtu 8950                  # 50B VXLAN overhead, underlay MTU 9000

auto br10101
iface br10101
    bridge-ports vxlan10101
    bridge-vids 10101
    bridge-vlan-aware yes

# FRR: EVPN advertises MAC/IP, not flood-and-learn
router bgp 65001
 neighbor SPINE peer-group
 neighbor SPINE remote-as 65001
 address-family l2vpn evpn
  neighbor SPINE activate
  advertise-all-vni

GRE vs. VXLAN: Choosing Encapsulation When Every Byte Costs

GRE is the oldest tool in this shed and I still reach for it first when I need a simple point-to-point tunnel. No control plane, no VNI, no multicast — just encapsulate and go. We use GRE tunnels extensively for origin fetch isolation and for shunting dirty traffic to scrubbing centers. Configuration is two lines, troubleshooting is `tcpdump -i gre0`, and the overhead is a predictable 24 bytes (20 byte outer IP + 4 byte GRE). Where VXLAN gives you a 50-byte header with UDP and VXLAN headers, GRE leaves more room for payload before you hit MTU issues. On our 1500-byte internet-facing paths, that 26-byte difference determines whether you fragment or not.

VXLAN wins when you need multi-tenancy and L2 extension at scale. GRE has a 32-bit key field but almost nobody implements it consistently, and you don't get a standardized control plane. VXLAN with EVPN gives us 16 million VNIs, per-VNI RT/RD isolation, host route mobility, and proper ARP suppression. Last year we migrated our Anycast log aggregation network from a mesh of GRE tunnels to VXLAN/EVPN. Overnight we cut broadcast storms by 90% because EVPN stopped flooding unknown unicast. If you're building isolated environments for customers, for different cache tiers, or for control-plane versus data-plane separation inside a PoP, VXLAN is the only sane option. For a 1:1 tunnel between two boxes, it's overkill.

The performance difference isn't what most blogs claim. On modern Intel Xeon and AMD EPYC with kernel bypass or DPDK, both encapsulations forward at line rate if you handle offloads correctly. The bottleneck is usually TSO/GRO and NIC checksum offloads, not the protocol itself. Per the latest guidance in HTTP Working Group Specifications and performance audits detailed on web.dev Articles, we tune MSS clamping rather than relying on PMTUD, because PMTUD breaks too often over the public internet. My rule of thumb: if you control the end-to-end MTU (inside your PoP or backbone), prefer VXLAN; if you cross unmanaged networks, GRE's smaller header saves you painful Fragmentation bugs.

# GRE tunnel for origin isolation - Linux iproute2
# No control plane, static endpoint, used for per-customer origin fetch

ip tunnel add gre-o101 mode gre remote 203.0.113.45 local 198.51.100.11 ttl 255
ip link set gre-o101 mtu 1476 up  # 1500 - 24B GRE overhead
ip addr add 192.0.2.1/30 dev gre-o101
ip route add 10.100.5.0/24 via 192.0.2.2 dev gre-o101 table 101

# MSS clamping to avoid PMTUD blackholes - critical for CDN origins
iptables -t mangle -A FORWARD -o gre-o101 -p tcp --tcp-flags SYN,RST SYN \
  -j TCPMSS --clamp-mss-to-pmtu

# Simple health check - withdraw route if GRE peer dies
# bfd peer 192.0.2.2 interval 300 multiplier 3

WireGuard at Scale: Where Encryption Meets Overlay Performance

We didn't adopt WireGuard because it was trendy. We adopted it because our previous IPsec mesh for inter-PoP control traffic was eating 30% of a core per 10 Gbps and required a full-time engineer just to manage IKE. WireGuard's crypto is boring in the best way: ChaCha20Poly1305, Curve25519, no cipher negotiation, and a 1-RTT handshake that actually completes. I remember our first test in 2021 — two bare-metal caches in Ashburn and Chicago pushing 18 Gbps of midgress traffic over a single WireGuard tunnel with less than 8% CPU overhead on a 16-core host. IPsec couldn't touch that without dedicated cards.

Scaling WireGuard isn't plug-and-play though. The upstream implementation is point-to-point and has no built-in key distribution. You can't manually sync 15,000 tunnel keys. We built a lightweight controller that watches our inventory service and pushes peer configs via Netlink — essentially WireGuard's `wg-quick` rewritten as a daemon with atomic reloads. Keys rotate every 6 hours, handshakes are staggered to avoid thundering herds, and we use a dedicated VRF for tunnel traffic so a rogue peer can't reach the management plane. One lesson we learned hard: don't run WireGuard inside a network namespace that shares the underlay's default route unless you enjoy routing loops. Keep the tunnel VRF separate and leak only what you need.

The other gotcha is MTU and performance tuning. WireGuard adds 60 bytes overhead (20 outer IP + 8 UDP + 32 crypto/auth), worse than both GRE and VXLAN. On our 9000-byte backbone we don't care, but on 1500-byte transit links we set tunnel MTU to 1420 and enforce MSS clamping to 1380. Also, WireGuard is single-threaded per peer in kernel mode — a single tunnel won't scale past one core. For 100+ Gbps between two super-PoPs we create 4-8 parallel tunnels and hash flows with ECMP, same principle we use for backbone links. That pattern gave us linear scaling to 80 Gbps per host pair without needing crypto accelerators.

From Lab to PoP: A Practical WireGuard + VXLAN Implementation

Theory's great until you're staring at a PoP router at 2 AM wondering why your overlay's flapping. In production CDN overlays, we rarely choose just one encapsulation — we stack them. The pattern that’s proven most resilient for us is VXLAN for segmentation and service isolation, carried inside WireGuard for encryption. You get the best of both worlds: VXLAN gives you 16 million VNIs to slice tenants, cache tiers, and staging environments without touching the underlay, while WireGuard gives you modern, kernel-accelerated encryption that doesn't melt your CPUs at 100Gbps.

Here's how we actually wire it. On each edge PoP server, we don't run VXLAN directly over the physical interface — we bind it to a WireGuard interface. That way, every VXLAN frame is encrypted before it hits the wire, and you don't need IPsec gateways or messy GRE key management. You'll want to pin WireGuard to specific cores with RPS/XPS and enable wg-quick with persistent keepalives at 15 seconds for NAT traversal. For orchestration, we've moved away from hand-rolling configs to using control-plane daemons that distribute VNI mappings via EVPN — which ties directly into Software-Defined Networking for CDN: Programmable Traffic Control for centralized policy. It’s not trivial to bootstrap, but once it’s automated, adding a new PoP is a 4-minute job instead of a 4-hour ticket.

The devil, as always, is MTU. A standard 1500-byte underlay MTU leaves you with ~1420 bytes of usable payload after you stack 50 bytes for VXLAN + UDP + IP and another 32 bytes for WireGuard. If you don't clamp MSS everywhere, you'll create black holes that only show up with large POSTs or video chunk revalidation. We set our overlay MTU to 1380 and enforce TCP MSS clamping to 1340 on every VTEP. It's a 5.3% overhead hit, but it's predictable — and predictable beats efficient but broken every time. This design also plays nicely with Network Function Virtualization in CDN Infrastructure: Virtual Appliances at Scale when you're running virtual cache nodes and WAFs inside the same overlay.

# Production pattern: VXLAN over WireGuard (Ubuntu 22.04 / kernel 6.2+)
# 1. Create WireGuard mesh interface (keys via Vault/age)
ip link add dev wg0 type wireguard
wg set wg0 private-key /etc/wireguard/privatekey peer $PEER_PUB allowed-ips 0.0.0.0/0 endpoint $PEER_IP:51820 persistent-keepalive 15
ip addr add 10.255.0.1/32 dev wg0
ip link set wg0 up mtu 1440

# 2. Create VXLAN VTEP bound to WireGuard IP, not physical NIC
ip link add vxlan100 type vxlan id 100 dstport 4789 \
  local 10.255.0.1 remote 10.255.0.2 dev wg0 nolearning
ip link set vxlan100 up mtu 1380

# 3. Bridge to service VRF or attach directly to cache container
ip link add br-cache type bridge
ip link set vxlan100 master br-cache
ip link set br-cache up

# 4. Clamp MSS to prevent fragmentation black holes
iptables -t mangle -A FORWARD -o wg0 -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
iptables -t mangle -A FORWARD -o vxlan100 -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1340

Troubleshooting at Scale: MTU, Key Rotation, and Control Plane Storms

When you've got 400+ PoPs and 12,000 WireGuard tunnels, your failure modes change completely. The first thing that’ll bite you is key rotation. WireGuard itself doesn't have built-in rekeying negotiation like IKEv2 — if you rotate a peer's public key by pushing a new config and restarting wg-quick, you'll drop ~150-300ms of traffic per tunnel during handshake. At scale, doing that sequentially across a full mesh takes hours. We've learned to do hitless rotation with two-peer entries: add the new key as a second peer with a higher priority, wait for the handshake to complete (check wg show latest-handshakes), then decommission the old key. Automate it, because manual rotation at 3 AM during an incident is how you lock yourself out of 40 PoPs.

The second killer is the control plane. If you're using EVPN/BGP to distribute VXLAN MAC/IP mappings, a single flapping VTEP can generate a storm of 20k updates per second and overwhelm your route reflectors. We saw this in 2023 when a top-of-rack switch with a bad optic started bouncing — within 90 seconds it had pushed 1.8M withdrawals that propagated to every PoP, causing 12% cache miss fill failures. Now we dampen aggressively: BGP route-flap dampening with a 15-minute half-life, EVPN route limits per VNI (capped at 4k MACs per VNI), and we separate the underlay BGP from the overlay BGP into different VRFs. Your choice of CDN Backbone Topology Design: Ring, Mesh, and Hierarchical Architectures directly dictates how painful that blast radius is — hierarchical designs contain storms better than a flat full mesh, even if they add 3-4ms of stretch.

Finally, don't trust ping for overlay health. ICMP will pass with a 64-byte payload while 1400-byte video segments are getting silently dropped due to PMTUD failure. Our health checks now send 1350-byte UDP probes with the DF bit set across every VNI every 10 seconds, and we track loss and jitter per tunnel. If loss on probes exceeds 0.5% for 2 minutes, we automatically withdraw that tunnel from the ECMP group. That one change cut our "mystery rebuffering" tickets by 62% last quarter. At this layer, observability isn't optional — export WireGuard handshake times, VXLAN FDB counts, and encapsulation drops to Prometheus and alert on rate-of-change, not absolute values.

Frequently Asked Questions