MPLS in CDN Backbone Networks: Label Switching for Predictable Performance

Back in 2019 we had a major live sports event push 40% more traffic than our capacity models predicted, and our IP-routed backbone between Ashburn and Frankfurt started doing what IP does best under stress — picking the shortest path for everyone and congesting it to death. Latency spiked from 72ms to over 190ms, we lost cache-fill traffic, and origin shields started timing out. That night is when we finally stopped treating MPLS as “something the telcos use” and rebuilt our entire CDN backbone around label switching.

Why Pure IP Routing Fails CDN Backbones at Scale

When you've run a CDN backbone for a decade, you learn that IP's greatest strength — distributed, hop-by-hop destination-based forwarding — is also its biggest liability for predictable performance. Every router makes an independent decision based on its IGP view, which means you can't really engineer traffic. I've watched our backbone repeatedly hammer the same low-cost link while a slightly longer path sat half-empty, simply because OSPF thought it was “better.” You can tweak metrics and try to hack it with ECMP Routing in CDN Networks: Equal-Cost Multi-Path Traffic Distribution, but ECMP only helps when paths are truly equal-cost, and in a global backbone they rarely are.

What we needed wasn't faster lookups, it was deterministic paths. MPLS gives you that by decoupling forwarding from IP destination lookups. Instead of every core router doing a longest-prefix match, we push a label stack at the ingress PE and core P routers just swap labels. It's not about speed per se — modern ASICs do IP lookup at line rate anyway — it's about control. Once I can pin a specific class of traffic to a specific Label Switched Path (LSP), I can guarantee that cache-fill, midgress replication, and client-facing delivery don't fight each other on the same links. Our shift wasn't instant, and we burned a few months learning the hard way that LDP alone won't save you if you don't pair it with proper traffic engineering.

# Ingress PE - RSVP-TE LSP for cache-fill traffic
# Cisco IOS-XR style config we run on backbone PEs
mpls traffic-eng
 interface GigabitEthernet0/0/0/0
  backup-path tunnel-te 100

interface tunnel-te 10
 ipv4 unnumbered Loopback0
 destination 192.0.2.45  # Egress PE in FRA
 path-option 1 explicit name VIA-AMS-LOW-LATENCY
 path-option 2 explicit name VIA-LON-BACKUP
 path-option 3 dynamic
 autoroute announce
 forwarding-adjacency holdtime 10
 signalled-bandwidth 20000  # 20 Gbps reservation
 priority 2 2
 record-route

explicit-path name VIA-AMS-LOW-LATENCY
 index 1 next-address strict ipv4 unicast 198.51.100.12
 index 2 next-address strict ipv4 unicast 198.51.100.18
 index 3 next-address strict ipv4 unicast 192.0.2.45

# Verify label bindings and RSVP session
# show mpls traffic-eng tunnels 10 brief
# show mpls forwarding labels 24012 detail

Designing Label Switched Paths for Predictable Cache Performance

There's a common misconception that MPLS is just about slapping labels on packets. If that's all you do, you haven't solved anything. The real work is in how you design your LSPs and what you map into them. We've settled on a model where we maintain three distinct LSP meshes: one for bulk cache-fill (high throughput, tolerant of slightly higher latency), one for interactive delivery and API traffic (low latency, strict jitter budget), and one for control-plane and purging (tiny but can't be dropped). Each gets its own RSVP-TE bandwidth reservation and explicit path. I don't trust unconstrained CSPF after what happened during that 2021 fiber cut between Chicago and New York — dynamic path computation kept flapping LSPs onto a congested backup link because it technically met constraints.

Now we use explicit paths for primary LSPs and keep dynamic as a last resort path-option, combined with Fast Reroute (FRR) facility backup for sub-50ms failover. Penultimate Hop Popping (PHP) helps keep egress PE overhead down, but we've had to disable it on a few nodes where we need precise QoS marking per label — you can't do per-label EXP-to-TC mapping if the label's already been popped. And if you're thinking about scaling this, consider how it interacts with newer path selection models like Segment Routing for CDN Traffic Engineering: Source-Routed Path Selection. We run SR-MPLS in our newer PoPs, but our core is still RSVP-TE because we need that hard bandwidth reservation and admission control. You can't get predictable performance if you can't say "no" when a path is full.

# LFIB and label stack inspection on P router
# Captured during peak - note per-EXP queuing

RP/0/RP0/CPU0:core-ams01# show mpls forwarding labels 24012 24024 detail
Local  Outgoing    Prefix             Outgoing     Next Hop        Bytes
Label  Label       or ID              Interface                    Switched
------ ----------- ------------------ ------------ --------------- ------------
24012  24024       tunnel-te10        Gi0/0/0/1    198.51.100.14   48392349234
  Label Stack (Top -> Bottom): { 24024  16045 }  # Transport + Service label
  EXP value: 4 (cache-fill) -> TC 2, Queue 3 (AF31)
  EXP value: 5 (interactive) -> TC 5, Queue 0 (EF)
  FRR: protected by tunnel-tp 101, ready (45ms switchover)
  TTL propagation: disabled for core hiding (ttl-propagate off)

# tcpdump style label visibility (RFC 3032)
# 00:00:0c:9f:f0:01 > 00:00:5e:00:01:02, MPLS (label 24024, exp 4, [S]0, ttl 255)
#                              (label 16045, exp 0, [S]1, ttl 254), IP ...

What Predictable Actually Looks Like on the Wire

Engineers love to talk about MPLS as if it's magic, but I measure it like anything else: jitter, loss, and convergence. After we segmented our backbone and moved to MPLS-TE, our p95 inter-PoP jitter dropped, and more importantly, it stopped correlating with total backbone utilization. Before, when we hit 65% utilization, jitter would explode. Now we push to 85% before we see the same degradation, because traffic isn't piling onto one IGP shortest path. We've also tied our Network Function Virtualization in CDN Infrastructure: Virtual Appliances at Scale edge nodes into the MPLS domain so virtual caches get the same deterministic paths as bare metal — that was a pain to get right with vPEs, but worth it.

Don't take my word for it — measure your own backbone with RFC 2544-style loss/latency tests per LSP, not just per link. We run continuous TWAMP probes inside each LSP's class and export EXP-aware telemetry via gNMI. The IETF specs matter here, and I still reference the HTTP Working Group Specifications when reasoning about how transport-layer decisions impact cache-fill behavior, plus web.dev Articles for what the client actually experiences at the edge. MPLS doesn't fix bad congestion control or poor origin plumbing, it just ensures the network between PoPs isn't the unpredictable part. The numbers below are from our US-EU backbone before and after we fully cut over to label switching for all inter-PoP traffic.

Putting LSPs Into Production: From RSVP-TE to Segment Routing

Here’s where most CDN teams get stuck — they lab MPLS perfectly, then try to forklift it onto a 40-PoP backbone carrying 8 Tbps at peak and wonder why cache-fill traffic starves eyeball delivery. Don’t do that. Start by carving your backbone into traffic-engineered classes. We typically run three LSP classes: Gold for cache-fill and origin shield pull (latency-sensitive, can’t tolerate loss), Silver for mid-tier cache-to-cache rebalancing, and Bronze for log pushes, purge propagation, and telemetry. Each gets its own explicit path and bandwidth reservation. If you’re still on RSVP-TE, you’ll be statically reserving — say, 30 Gbps for Gold on a 100G backbone link, with auto-bandwidth adjusting every 300 seconds. If you’ve moved to SR-MPLS (and you should if you’re building new in 2025-2026), you get the same traffic steering without the RSVP state hell — no more 15,000 LSP states melting your old Catalysts.

Your label mapping strategy matters more than the signaling protocol itself. For CDN workloads, you don’t want per-flow entropy — you want per-service predictability. We map incoming DSCP + PoP role to MPLS EXP bits at the ingress LER. It’s simple and it works. DSCP AF41 (your video manifests and small objects) -> EXP 5, DSCP AF31 (large video chunks) -> EXP 3, background sync -> EXP 1. That EXP then drives your weighted queuing all the way across the core, so even when a Frankfurt-to-Warsaw link hits 87% utilization during a Champions League final, your Gold LSP isn’t dropping packets behind a log flood. Pair this with CDN Backbone Topology Design: Ring, Mesh, and Hierarchical Architectures — if you’re on a hierarchical core, put your primary Gold LSPs on the low-latency mesh and keep Bronze on the cheaper transit rings.

And please, automate your LSP provisioning. Hand-configuring explicit paths is how you get a 2 AM outage because someone fat-fingered a next-hop. We’ve standardized on a controller-driven model where the PCE computes diverse paths and pushes them via PCEP/BGP-LS. It’s the same philosophy behind Software-Defined Networking for CDN: Programmable Traffic Control, just applied to label stacks instead of flow tables. Our PCE recalculates every 30 seconds using live telemetry (link delay via TWAMP, queue depth via gNMI) and will preemptively reroute a Gold LSP if delay spikes by >4ms or loss exceeds 0.01%. That’s how we keep p95 origin fetch time under 85ms even when a backbone fiber cut takes out 400G between two super-PoPs. Here’s a stripped-down SR-MPLS config snippet we use on IOS-XR ingress routers for a cache-fill LSP:

segment-routing
 traffic-eng
  policy GOLD-CACHE-FILL-01
   color 100 end-point ipv4 10.255.0.11
   candidate-paths
    preference 100
     dynamic
      metric type latency
      constraints
       bandwidth 30000  ! 30G reserved for cache-fill
       affinity exclude red  ! avoid high-loss links
     !
    !
    preference 80
     explicit segment-list LOW-LATENCY-BACKUP
      index 10 mpls label 16005  ! Frankfurt Core
      index 20 mpls label 16012  ! Prague Transit
      index 30 mpls label 16011  ! Warsaw Egress
     !
    !
  steering
   bgp color 100  ! auto-steer BGP routes with color 100
  !
 !
! Map DSCP to EXP at ingress
class-map match-any CDN-GOLD
 match dscp af41 af42
policy-map INGRESS-LER-MARKING
 class CDN-GOLD
  set mpls experimental topmost 5
!

Advanced Patterns: Fast Reroute, Entropy, and Debugging Label Chaos

Once your basic LSPs are stable, that’s when the fun edge cases start. The big one for CDNs is micro-loops during reconvergence. Pure IP FRR can take 600-900ms to converge across a 50-node backbone — an eternity when you’re serving 200k concurrent HLS sessions. With MPLS Fast Reroute (FRR), we’ve got detour LSPs pre-signaled and ready. Facility backup with bypass tunnels gives us <50ms switchover, verified in prod — we cut a 100G link between Amsterdam and London during peak and saw exactly 38ms of loss on Gold traffic, zero player rebuffers. But don’t just enable FRR and forget it. You’ve got to validate your detour bandwidth. We’ve seen Bronze bypass tunnels suddenly carry Gold traffic after a double failure, saturate, and then you’re dropping cache-fill anyway. Reserve at least 40% headroom on every bypass.

The other gotcha is load balancing. MPLS doesn’t do per-packet magic by default; if you don’t handle entropy right, one member of your 4x100G LAG will sit at 92% while the others idle at 40%. You need entropy labels (RFC 6790) or proper flow-aware hashing on the LSRs. We enable entropy label capability on all P-routers and push an entropy label for any LSP carrying >10k flows. That dropped our max link utilization variance from 34% to under 8% across the core. And when debugging, stop looking at IP traceroute — it lies under MPLS. Use traceroute mpls and show mpls forwarding to see the actual label stack. If you’re seeing inconsistent latency between PoPs, check for MTU issues first. An extra 4-byte label stack on top of a 1500-byte cache object plus VXLAN can silently fragment. We standardize on 9000-byte jumbo end-to-end and set mpls mtu 9100 on every core interface, which eliminated the 5-7% throughput collapse we saw on 1MB+ objects.

For long-term operations, tie your LSP telemetry directly into your PoP health score. We export LSP delay, jitter, and loss via gRPC every 10 seconds and feed it to our PoP selection logic. If the LSP from Munich to Milan shows sustained jitter >2ms for 60 seconds, we de-preference Milan for cache-fill and shift to Zurich — no human intervention. This tight coupling between backbone transport and edge routing is why you need to read CDN PoP Interconnection Design: Building Reliable Edge-to-Core Links before you tune LSPs in isolation. Your interconnection design dictates your label edge — a PoP with dual diverse 100G uplinks can support true 1:1 LSP protection, while a single-homed micro-PoP is stuck with best-effort. Don’t promise 99.99% predictable delivery if your physical diversity is 99.5%.

Frequently Asked Questions