VLAN Design for CDN Edge Infrastructure: Logical Network Separation

Back in 2019 we had a PoP in Frankfurt drop off the map for thirteen minutes because a single misbehaving cache node flooded the management network with multicast. It wasn't a hardware failure — we'd put IPMI, BGP sessions, and client-facing transit traffic all in the same flat /20. That night taught me more about VLAN design than any certification ever did, and if you're building CDN edge infrastructure, logical separation isn't optional, it's survival.

Separating the Three Planes You Can't Afford to Mix

When I walk into a new edge PoP design review, the first thing I'm looking for is how they've split their planes. In a CDN edge, you've got at least three distinct traffic types fighting for the same wires: the management plane (SSH, SNMP, IPMI/iDRAC, telemetry), the control plane (BGP, OSPF/IS-IS, health checks, anycast signaling), and the data plane (eyeball traffic in, cache fill from origin or parent, and inter-cache traffic). Put them together and you don't just get a security problem, you get a noisy neighbor problem where a cache fill burst can starve your BGP keepalives. I've seen it happen, and it's ugly to debug at 2 a.m.

Our standard approach now is boring on purpose, which is exactly what you want. We assign dedicated VLANs per function and never reuse a VLAN ID across PoPs for the same function — VLAN 100-199 for management, 200-299 for control, 300-399 for data. That makes automation and troubleshooting predictable. Management VLANs are always routed through a dedicated VRF and firewalled at the aggregation layer, control VLANs get strict ACLs that only permit routing protocol traffic and monitoring, and data VLANs are split even further: one for internet peering/transit, one for origin fetch, and one for east-west cache peering. It sounds like overkill until you need to run ECMP Routing in CDN Networks: Equal-Cost Multi-Path Traffic Distribution cleanly without your hashing being polluted by unrelated flows.

One mistake I don't make anymore is routing between these VLANs on the top-of-rack switches. Let the ToRs be L2 only. Push all inter-VLAN routing up to the aggregation or spine pair where you have proper ACLs, CoPP, and visibility. That way a compromised cache node in the data VLAN can't directly pivot to your management network even if it guesses the subnet. The HTTP Working Group Specifications won't save you if your origin fetch VLAN can talk to your BMC network — you need that isolation enforced in hardware.

! ToR L2 VLAN provisioning - standardized across all edge PoPs
vlan 110
 name MGMT_OOB
vlan 210
 name CTRL_BGP_HEALTH
vlan 310
 name DATA_EYEBALL_TRANSIT
vlan 320
 name DATA_CACHE_FILL
vlan 330
 name DATA_CACHE_PEERING

interface range TenGigabitEthernet1/0/1 - 24
 switchport mode access
 switchport access vlan 310
 spanning-tree portfast
 spanning-tree bpduguard enable
 ip access-group CACHE-DATA-IN in

interface Vlan110
 description MGMT - no SVI on ToR, L2 only
 no ip address
!

Trunks, MTU, and Pruning: Making VLANs Scale at 100G

You can draw beautiful VLAN diagrams, but they fall apart on the trunk. At the edge we've standardized on dual-homed ToRs with 2x 100G uplinks to the spine in an LACP bundle, and every VLAN we just defined has to ride that trunk correctly. Early in my career I left the default `switchport trunk allowed vlan all` on and paid for it when someone accidentally extended a test VLAN across the entire PoP and created a spanning-tree loop that took down the cache peering network. Now we prune aggressively — only the VLANs needed on that rack ride the trunk, and new VLANs are explicitly allowed via automation.

MTU is the other detail that bites people. Your eyeball-facing data VLAN should stay at 1500 bytes — you don't control the internet, and fragments will kill your performance. But your origin fetch and cache peering VLANs? Those are inside your infrastructure, and you want 9000 byte jumbo frames there all day long. If you don't separate those into different VLANs, you can't set a per-VLAN MTU cleanly, and you'll end up clamping everything to 1500 and wasting 15-20% of your inter-cache throughput on overhead. We also set the trunk MTU to 9216 to account for the 802.1Q tag — forget that and you'll get silent drops on tagged frames that are brutally hard to catch without checking the counters on the ASIC.

This is also where good VLAN design interacts with the backbone. When you hand off from the edge PoP to the core, you don't want VLAN sprawl leaking into your backbone. We terminate all edge VLANs at the aggregation layer and map them to either VXLAN or MPLS in CDN Backbone Networks: Label Switching for Predictable Performance transport for anything that needs to go inter-PoP. That keeps your edge broadcast domains small — ideally under 48 ports per VLAN — and your backbone clean. The docs at web.dev Articles are great for frontend performance, but at the network layer, keeping broadcast and unknown unicast contained is what actually preserves your tail latency.

! Spine-facing trunk - explicit pruning and MTU
interface Port-channel10
 description UPLINK_TO_SPINE_AGG
 switchport mode trunk
 switchport trunk allowed vlan 110,210,310,320,330
 mtu 9216
 lacp rate fast

interface Ethernet1/1
 channel-group 10 mode active
interface Ethernet1/2
 channel-group 10 mode active

! Per-VLAN MTU policy applied at SVI on aggregation
interface Vlan320
 description DATA_CACHE_FILL - JUMBO
 mtu 9000
 ip address 10.32.0.1/24
!
interface Vlan310
 description DATA_EYEBALL - STANDARD
 mtu 1500
 ip address 203.0.113.1/24
!

What Good Isolation Actually Buys You: Measured Impact

For a long time I justified VLAN separation with hand-wavy arguments about "security best practices." That doesn't fly with finance or with engineers who have to manage 400 VLANs. So we started measuring it. After we segmented a legacy PoP in Amsterdam that was running a /19 flat network, we saw immediate changes in failure domains and operational load. The numbers below aren't theoretical — they're from our own Grafana and our incident postmortems over six months before and after the re-design, and they're why we don't debate this anymore.

The biggest win isn't even security, though that's critical. It's blast radius and MTTR. In a flat network, a broadcast storm, a spanning-tree TCN, or a bad optic spewing errors hits everything. With proper VLAN pruning, that storm is isolated to one function on one rack pair. Our NOC can now pinpoint the VLAN from the alert alone and shut just that trunk member without touching client traffic. We've cut our mean time to isolate a network fault from over an hour to minutes, and that's without even touching more advanced techniques like Segment Routing for CDN Traffic Engineering: Source-Routed Path Selection which we run at the backbone layer.

You'll also notice the performance side. When cache-fill (often 50-80 Gbps per rack during peak revalidation) isn't contending with BGP control traffic, your session flaps disappear and your p99 cache fill time tightens up. That's directly tied to origin response time and, ultimately, cache hit ratio at the edge. Isolation gives you predictable performance under load, which is the whole point of a CDN.

Deploying It for Real: A Zero-Downtime VLAN Rollout Template

Diagrams don't forward packets. When we roll VLAN separation into a live edge PoP — typically 12 to 20 cache servers, 2x leaf switches, and a pair of routers — we start with a boring spreadsheet and stick to it religiously. You've got 4094 IDs to play with, but you'll regret random assignments six months from now. What works: VLAN 10-99 for management/IPMI and OOB, 100-199 for control plane (BGP, anycast health checks, automation), 200-299 for origin/mid-tier fetch, 300-399 for client-facing delivery, and 900-999 reserved for quarantine/scrubbing. Document the native VLAN explicitly as 999 and never use VLAN 1. At one EU PoP we inherited, VLAN 1 was still carrying untagged monitoring traffic and it took a single misconfigured trunk to blackhole 18% of requests for 11 minutes.

Don't try to migrate everything in one maintenance window — that's how you get a 3 a.m. rollback. We pre-provision the new VLANs with pruning disabled, push config via Ansible, then move hosts port by port. Validate each move with show interfaces trunk and an MTU probe before you cut traffic. If you're virtualizing cache or WAF nodes on the same hardware, this is where Network Function Virtualization in CDN Infrastructure: Virtual Appliances at Scale really pays off — you can pin vNICs for control, data, and management to distinct VLANs at the hypervisor level without recabling. And if you're already running a centralized controller, tie the VLAN provisioning into it; Software-Defined Networking for CDN: Programmable Traffic Control makes it trivial to push consistent templates across 50 PoPs instead of hand-editing each leaf.

! Arista EOS - Edge Leaf Template (MLAG pair)
vlan 110
   name MGMT_OOB
vlan 120
   name CTRL_BGP_HEALTH
vlan 210
   name ORIGIN_FETCH
vlan 310
   name CLIENT_DELIVERY
!
interface Ethernet1-20
   description Cache Servers - Access Ports
   switchport mode access
   switchport access vlan 310
   mtu 9216
   spanning-tree portfast
!
interface Ethernet49-50
   description Uplinks to Spine - 100G Trunk
   switchport mode trunk
   switchport trunk allowed vlan 110,120,210,310
   switchport trunk native vlan 999
   mtu 9216
!
interface Vlan110
   description Management SVI
   ip address 10.255.110.1/24
   ip virtual-router address 10.255.110.254
!
! Verify: show interfaces trunk | show vlan brief | ping mtu 9000

Advanced Patterns and When VLANs Leak

Once you've got the basics stable, you'll hit the 4094 VLAN ceiling or the need to stretch isolation beyond a single rack. For multi-tenant edges where you terminate different content providers on shared hardware, Q-in-Q (802.1ad) is a lifesaver — you push a customer S-TAG at the leaf and keep your internal C-TAGs untouched. We've used it to isolate purge APIs and logging streams per tenant without burning a new VLAN per customer on the spine. For PoPs that need L2 adjacency across sites — say, for anycast failover or live migration — don't stretch VLANs with old-school VPLS. Use EVPN/VXLAN with a proper L3 underlay. It gives you logical separation with control-plane learning, and you avoid the spanning-tree nightmares that come with extending broadcast domains across your CDN Backbone Topology Design: Ring, Mesh, and Hierarchical Architectures. One mesh backbone we migrated saw BUM traffic drop 73% after moving from stretched VLANs to EVPN.

When isolation fails, it's almost never a mysterious bug — it's three things. First, native VLAN mismatch. If leaf A thinks native is 999 and leaf B is still on 1, untagged control frames will hop VLANs and you'll see intermittent BGP flaps that look like packet loss. Second, forgotten allowed vlan — we pruned VLAN 210 from a spine trunk to "save bandwidth" and then spent two hours wondering why origin fetches were timing out with 100% packet loss on that rack only. Always keep show interfaces trunk in your post-change checks. Third, MTU blackholing. If your client-facing VLAN is 1500 but your origin VLAN is 9000 and you forgot to set mtu 9216 on the trunk, you'll get silent fragmentation failures — large objects (we saw this with 4K video segments over 6MB) will stall while small thumbnails fly through. Use ping size 9000 df-bit on every VLAN after provisioning.

Pro tip: instrument per-VLAN counters from day one. sFlow or IPFIX sampled at 1:4096 per VLAN will tell you within seconds if your scrubbing VLAN is suddenly carrying 40 Gbps of client traffic or if your control plane VLAN is seeing broadcast storms (we alert if broadcast exceeds 2% of link capacity for more than 90 seconds). That visibility is what turns VLANs from a neat diagram into actual operational isolation. You don't want to find out your separation failed during a DDoS — you want a dashboard screaming at you before it does.

Frequently Asked Questions