CDN PoP Interconnection Design: Building Reliable Edge-to-Core Links

At 2:13 AM on a Saturday last winter, I watched our Santiago PoP drop off the core because a single edge-to-core bundle saturated and BFD timed out faster than BGP could reconverge. It wasn't a fiber cut or a router failure — just a poorly balanced interconnection that'd been running hot for weeks. That night forced us to rethink how we build PoP interconnection links, and if you've run a CDN edge for any length of time, you've probably had a similar wake-up call.

Why Edge-to-Core Links Break Under Real Traffic

Most PoP designs look solid on a whiteboard. You've got your edge caches, your aggregation switches, and a couple of 100G links back to the regional core. On paper, there's enough capacity with N+1 redundancy. In practice, though, edge-to-core traffic isn't smooth or predictable. Cache misses spike together, revalidation storms hit after an origin flap, and suddenly your steady 40% utilization jumps to 95% for ten minutes. I've seen this pattern kill PoPs that passed every capacity review because the reviews assumed average throughput, not p95 burst behavior during failure scenarios.

The other problem is how we think about the link itself. A PoP interconnection isn't just a cable between two routers — it's a failure domain that includes optics, patch panels, DWDM transponders if you're on dark fiber, and the control plane that decides when to use it. We learned the hard way that you can't treat the physical path and the routing policy as separate problems. For years we ran two diverse fibers into the same aggregation chassis, which meant a firmware bug took out both links at once. Now we insist on chassis, power, and path diversity, and we validate it with traceroutes and optical loss measurements, not just diagrams. If you're building on an MPLS backbone, you'll want to understand how MPLS in CDN Backbone Networks: Label Switching for Predictable Performance handles label-switched paths differently than pure IP peering when those links flap.

We also stopped trusting default timers. Default BGP hold times and BFD intervals that work fine in a lab will cause cascading reroutes when your edge-to-core links are congested. We tune BFD to 300ms x 3 for inter-PoP links and run BGP graceful restart so a single control-plane hiccup doesn't withdraw thousands of prefixes. Here's the baseline we deploy on every new PoP edge router — it's not fancy, but it's kept us from having that 2 AM page again:

! PoP Edge-to-Core Interconnect - Baseline
interface HundredGigE0/0/0/0
 description P2C-BUNDLE to Core-01 (Primary)
 carrier-delay up 2000 down 500

interface HundredGigE0/0/0/1
 description P2C-BUNDLE to Core-02 (Secondary)
 carrier-delay up 2000 down 500

router bgp 64512
 bfd map ipv4 10.255.0.1 10.255.0.2 interval 300 min_rx 300 multiplier 3
 neighbor 10.255.0.1 remote-as 64512
  address-family ipv4 unicast
   maximum-prefix 100000 95 restart 300
  bfd fast-detect
 !
 neighbor 10.255.0.2 remote-as 64512
  address-family ipv4 unicast
   maximum-prefix 100000 95 restart 300
  bfd fast-detect

! Track link bundle health
bfd session 10.255.0.1
 bfd profile CORE-LINKS
!

Designing Redundant PoP Interconnects That Don't Waste Capacity

Redundancy without efficiency is just burning money. I've audited PoPs where they bought four 100G links and ran them active-passive, so 75% of capacity sat idle 99% of the time. The finance team loved the reliability numbers, but we couldn't justify the cost per delivered gigabit. A better approach is active-active with proper load sharing, but that only works if your hashing and path selection actually spread flows evenly. We use a mix of resilient hashing on the edge and flow-aware ECMP in the core so elephant flows from large object delivery don't pin a single link while the others sit idle.

For most regional designs, we've moved away from simple ECMP to a more controlled model. ECMP is great inside the PoP — you can read more about tuning it in ECMP Routing in CDN Networks: Equal-Cost Multi-Path Traffic Distribution — but for edge-to-core, we want deterministic failover and traffic engineering. We run two physically disjoint paths (often different carriers or different fiber sheaths from the same carrier) and treat them as unequal-cost paths with BGP local-preference and MPLS TE metrics. One link carries 60-70% of traffic under normal conditions because it has lower latency; the second takes the rest and absorbs the full load if the primary degrades. That asymmetry sounds odd, but it means we're constantly proving both paths work with real traffic, instead of discovering a dead standby link during an outage.

Segment routing has made this a lot cleaner. Instead of RSVP-TE tunnels that require state on every hop, we can push a policy from the edge that says "prefer this PoP interconnection unless latency exceeds 25ms, then shift." It integrates nicely with our automation — when our monitoring sees packet loss creeping past 0.05% on a link, we adjust the SR policy to drain it before users notice. If you haven't explored this, Segment Routing for CDN Traffic Engineering: Source-Routed Path Selection covers the fundamentals we built on. We also align our HTTP stack tuning with specs from the HTTP Working Group Specifications and performance guidance from web.dev Articles so retransmits and connection coalescing don't amplify a small network issue into a user-visible one. Here's a simplified SR-TE policy we use to steer cache-fill traffic away from congested edge-to-core links:

! SR-TE Policy for Edge-to-Core Steering
segment-routing
 traffic-eng
  policy CACHE-FILL-CORE-01
   color 100 end-point ipv4 192.0.2.1
   candidate-paths
    preference 100
     explicit segment-list PRIMARY-PATH
      index 10 sid 16002   ! Core-01 Node SID
      index 20 sid 24100   ! Low-latency link SID
     !
    preference 80
     explicit segment-list SECONDARY-PATH
      index 10 sid 16002
      index 20 sid 24200   ! Secondary link SID
     !
    constraints
     latency max 25000 us  ! Auto-fallback if >25ms
     bandwidth 80000 mbps
  !
  steering
   bgp color 100
   up-delay 10000  ! Wait 10s after recovery
!

What We Actually Measure on Edge-to-Core Links

You can't fix what you don't measure, and interface counters aren't enough. We poll every PoP interconnection for four things at 30-second granularity: one-way latency via TWAMP, packet loss via synthetic probes, link utilization per direction, and BGP session stability. The first two tell us about user impact before counters spike; the last two tell us about capacity risk. I remember arguing with a vendor who insisted 0.1% loss was "acceptable" — it might be on a backbone, but on an edge-to-core link serving cache misses, that loss triggers TCP backoff and collapses throughput for large objects by 30-40%.

Our SLOs are deliberately tighter than carrier SLAs. We alert at 15ms one-way latency increase over baseline, not absolute latency, because baselines differ by geography. A 35ms link in São Paulo is normal; a 35ms link in Amsterdam means something's routing the long way around. We also track hash polarization — if ECMP is sending 80% of flows down one member link, we'll see it in flow telemetry even when SNMP says links are balanced by bytes. That distinction has saved us twice when a bad hash seed or an uneven 5-tuple distribution caused silent congestion.

The table below shows the real-world ranges we see across our fleet and what we use for capacity planning. These aren't vendor datasheets; they're 30-day p95 numbers from production PoPs with mixed traffic (video, small objects, API). Use them as a starting point for your own thresholds, but measure your own baseline — your edge to core mix will be different depending on cache hit ratio and origin distance.

Practical Implementation: Turning Redundant Designs Into Running Config

Diagrams with dual diverse paths look great on a whiteboard. Making them survive a 3 AM fiber cut is all about the boring details. In production, our standard edge PoP now lands with at least 2x 100G DWDM lambdas to two different core PoPs, plus a lower-capacity 10-40G IP transit or IX backup that we keep hot but de-preferenced. We've learned not to trust a single chassis — each link terminates on a different router and, where budget allows, different line cards. That way a bad optic, a failed ROADM, or a software crash doesn't take both directions down. If you're virtualizing any of those edge functions, you'll want to read how we handle stateful failover in Network Function Virtualization in CDN Infrastructure: Virtual Appliances at Scale — vCGNAT or vWAF instances can't just flap without dropping sessions.

Automation is what keeps this consistent across 60+ PoPs. We don't hand-configure BGP anymore. Every edge router gets templated from a central controller that pushes the same baseline: BGP with BFD at 300ms x 3 for core links (that's 900ms detection, fast enough without flapping on jitter), ECMP with consistent hashing, and explicit communities to steer cache-fill vs. eyeball traffic. For newer PoPs we've moved that templating to an SDN controller so we can re-weight paths in seconds instead of waiting for IGP convergence. The approach is covered in detail in Software-Defined Networking for CDN: Programmable Traffic Control, and it’s saved us more than once when we needed to drain a link for maintenance without announcing a withdrawal globally.

The trick most teams miss is testing the failure mode before traffic does. We run scheduled “chaos” drains every month — shut a core-facing interface, kill a BGP session, bounce BFD — and measure actual reconvergence. You’d be surprised how often a perfectly valid backup path won’t carry load because someone left a policer at 30G from initial turn-up. Here's the stripped-down FRR template we deploy on every edge-to-core session:

interface et-0/0/0 # Primary 100G to Core-A
  mtu 9200
!
interface et-0/0/1 # Diverse 100G to Core-B
  mtu 9200
!
router bgp 64512
 bgp router-id 10.254.0.11
 neighbor CORE-A peer-group
 neighbor CORE-A remote-as 64500
 neighbor CORE-A bfd profile FAST-BFD
 neighbor CORE-A addpath-tx-all-paths
 neighbor CORE-A timers 10 30
 !
 neighbor 10.255.0.1 peer-group CORE-A
 neighbor 10.255.0.2 peer-group CORE-A # Core-B
 !
 address-family ipv4 unicast
  maximum-paths 4
  bgp bestpath as-path multipath-relax
  bgp additional-paths select all
 exit-address-family
!
bfd profile FAST-BFD
 detect-multiplier 3
 receive-interval 300
 transmit-interval 300
!
route-map EDGE-OUT permit 10
 set community 64512:100 64512:3000 # region:role edge
 set local-preference 150
!

Advanced Patterns and Troubleshooting When Failover Looks Good But Isn't

The nastiest outages aren't total link loss — they're degraded links that stay up. We had a case last year where a dirty fiber between Madrid edge and our Paris core caused 0.8% packet loss. BGP stayed up, BFD at 300ms didn't trip, and ECMP happily sprayed 25% of cache-fill traffic into a black hole. P95 TTFB jumped from 42ms to 310ms before our alerts fired. Now we don't rely on liveness alone. Every edge-to-core bundle runs TWAMP or RFC 5357 probes at 100ms intervals, plus we track retransmit rates from our nginx/ATS logs. If loss exceeds 0.1% for 30 seconds or jitter variance doubles, we automatically de-preference that link via community, even if BGP is still established. That active de-prioritization beats waiting for a hard down.

Another classic gotcha is ECMP polarization and microburst collapse. With 4x 100G LAGs hashing on the usual 5-tuple, you’ll see one member sitting at 92G while the others sit at 58G — not because traffic is unbalanced, but because elephant cache-fill flows (we see 10-20G single flows during origin shield sync) hash to the same link. Add a failover that suddenly crams 2 links onto 1, and buffers overflow in 50-100ms bursts no 5-second SNMP poll will ever catch. We fix this with resilient hashing (consistent hash with flowlet switching where supported) and by decoupling link capacity from advertised IGP metrics. Instead of letting ISIS cost drive it, we use BGP link-bandwidth communities so the ingress edge knows the real 400G vs. 200G capacity. For how we choose when to use rings vs. partial mesh to avoid these bottlenecks in the first place, see CDN Backbone Topology Design: Ring, Mesh, and Hierarchical Architectures.

When you’re troubleshooting, start outside the router. Check optics first — `show interfaces extensive | match optics` will show you pre-FEC BER. If you’re seeing 1e-5 before FEC correction, you’re one dB away from disaster. Then verify control plane vs. data plane: we’ve chased “BGP converged” ghosts where FIB programming lagged by 4-7 seconds due to a slow line card. Our runbook now is: 1) TWAMP loss/jitter, 2) optic diagnostics and interface drops, 3) `show bfd session` and `show bgp summary` timestamps, 4) data-plane traceroute with MPLS labels visible. If all else fails, capture at both ends — we keep a 10-second ring buffer of sampled flow data on every edge router. It’s caught more silent MTU and ECN blackholes than any SNMP graph ever will.

Frequently Asked Questions