Back in late 2017, we had a transit provider flap for 47 minutes between Frankfurt and Warsaw. It wasn't a huge outage — just intermittent loss north of 2% — but our cache fill traffic was still riding their network, and end users in Poland saw rebuffering spike by 400%. That night I learned what every CDN operator eventually learns: when you don't own the path between your PoPs, you don't really control your quality of service. For a small CDN that lesson can wait, but past a certain scale it becomes an existential question for your CDN infrastructure.
When Transit Starts Hurting More Than It Helps
For the first few years, buying transit just makes sense. You pay someone else to handle long-haul capacity, you get global reach on day one, and your team can focus on caching and edge logic instead of fiber leases. I've built two networks that started that way, and there's no shame in it. A good transit network gives you 10, 40, even 100 Gbps in major metros for a predictable monthly fee, and with proper peering you can keep egress costs manageable while you find product-market fit.
The pain doesn't show up as a single bill. It shows up as death by a thousand cuts. Your mid-mile latency becomes whatever your transit provider's hot-potato routing decides that day. A 30 ms path between Madrid and Paris suddenly becomes 65 ms because your provider decided to hairpin through London to balance their own backbone. You file a ticket, they reroute it three days later, and in the meantime your origin shield can't keep up. I've seen teams try to solve this with more transit providers, and it helps for a while, but you're still playing someone else's traffic engineering game. You can't implement the kind of deterministic routing you'd get from MPLS in CDN Backbone Networks: Label Switching for Predictable Performance when the core isn't yours, and you certainly can't do fine-grained path control without owning the routers in the middle.
There's also a capacity planning problem that's easy to miss until you're in it. Transit pricing is still largely 95th percentile, and CDN traffic is bursty by nature. When we had a major software update go out for a customer — about 60 Gbps of unexpected fill across Central Europe — our transit bill that month jumped 38% even though our average throughput only rose 11%. If that fill had stayed on a private backbone, the marginal cost would have been near zero because we'd already lit the capacity. That's the moment most teams start seriously modeling network ownership.
# Real traceroute we saw during that Frankfurt -> Warsaw incident
# Transit path (unpredictable, 14 hops)
traceroute to 185.199.12.45 (Warsaw PoP) from Frankfurt
5 ae-5.core1.fra.transit-provider.net (80.81.192.20) 12.4 ms
6 ae-1.core2.ams.transit-provider.net (80.81.192.45) 28.7 ms # backhaul to Amsterdam
7 ae-3.core1.lon.transit-provider.net (80.81.192.89) 42.1 ms # then to London
8 ae-2.core1.waw.transit-provider.net (80.81.192.112) 64.8 ms
# Desired private backbone path (6 hops, consistent)
# fra01 -> ber01 -> waw01 on owned DWDM, ~18ms RTT stable
What Owning Your Backbone Actually Means in Practice
People hear "private backbone" and picture owning dark fiber across continents. Most CDNs don't do that, at least not at first. What you're really building is a set of long-haul wavelengths or leased lit capacity between your own routers, so your inter-PoP traffic never touches the public internet. My first backbone build was just four 100G waves: Ashburn to Chicago to Denver to San Jose. We leased the optics, colocated our own Juniper MX routers, and ran IS-IS inside. The whole project took nine months and cost more than any of us wanted to admit to finance, but overnight our cache fill success rate went from 99.2% to 99.92%.
Ownership changes how you think about routing entirely. On a transit network, you're essentially stuck with BGP best-path selection and whatever communities your provider respects — if they respect them at all. On your own backbone, you can run proper traffic engineering. You can keep latency-sensitive manifest and API traffic on your lowest-latency lambda, while bulk cache fill takes a cheaper, slightly longer path. You can use Segment Routing for CDN Traffic Engineering: Source-Routed Path Selection to pin specific customer traffic to specific paths without the operational overhead of RSVP-TE, and you can use ECMP Routing in CDN Networks: Equal-Cost Multi-Path Traffic Distribution properly at each core node because you control the hash and the next-hops.
That control also matters for debugging. When you've got your own optical layer, you know exactly where loss is happening. I'll never forget isolating a CRC error to a single dirty patch panel in Prague because our backbone telemetry showed a clean jump in pre-FEC errors on that span — something we would have never seen if we'd been looking at a transit provider's black-box MPLS core. The specs we rely on for HTTP delivery behavior are still the HTTP Working Group Specifications, but how quickly and reliably those HTTP responses move between your own PoPs is entirely a network ownership question. And for what users actually experience at the edge, the guidance in web.dev Articles on TTFB and LCP only translates to real-world performance if your mid-mile isn't adding random jitter.
! Simplified backbone router config - private backbone with SR-TE
router isis backbone
net 49.0001.1921.6800.1001.00
segment-routing mpls
traffic-eng level-2
interface HundredGigE0/0/0/0
description WAVE fra01 -> waw01 (owned 100G, 850km)
ipv4 address 10.255.0.1/30
isis circuit-type level-2
isis metric 15 ! low metric = preferred for low-latency traffic
segment-routing
traffic-eng
policy LOW_LATENCY_VIDEO
color 10 end-point ipv4 10.255.0.14
candidate-paths
preference 100
explicit segment-list FRA-WAW-LOWLAT
index 10 mpls label 16002 ! ber01 SID
index 20 mpls label 16005 ! waw01 SID
How to Decide: Cost, Scale, and Performance Trade-offs
So when should you actually build? I've made the call both ways, and the rule of thumb I use now is pretty simple: if your inter-PoP traffic consistently exceeds 30-40% of your total transit commit, and you're seeing more than 15 ms of unnecessary latency inflation on key PoP-to-PoP paths, it's time to run the numbers. That doesn't mean you flip a switch and replace all transit. Most mature CDNs I know run a hybrid — private backbone for east-west fill, replication, and shield traffic, plus peering and transit for last-mile delivery. The backbone is about protecting your origin-to-edge path; you still need eyeball networks to hand off to users.
The math is less scary than it looks once you break it down. A 100G wave on a busy corridor like Frankfurt-Vienna-Warsaw might cost you $6k-$9k per month if you're leasing lit capacity, plus router amortization and cross-connects. Compare that to paying $0.35-$0.80 per Mbps for transit at 95th percentile when you're pushing 80 Gbps of fill. At sustained scale, the per-gigabit cost on your own fiber drops to a tenth of transit. But — and this is important — if you're only moving 5-10 Gbps between regions, transit will be cheaper for a long time. Don't build a backbone because it feels more "serious." Build it because your usage pattern makes the CapEx pay back within 12-18 months.
Below is the comparison I wish someone had shown me before our first build. These aren't lab numbers; they're averages from a mid-size CDN I consulted for in 2023-2024 handling ~200 Tbps of monthly egress, measured across EU PoPs before and after they lit their first backbone triangle.
| Metric (EU Inter-PoP) | Transit Network | Private Backbone | Delta |
|---|---|---|---|
| p95 latency FRA-WAW (RTT) | 48 - 72 ms (varies) | 18.2 ms (stable) | -62% avg, 85% less jitter |
| Packet loss (24h avg, fill traffic) | 0.08% - 0.45% | 0.001% - 0.008% | ~50x reduction |
| Cost per sustained Gbps (fill) | $420 - $680 / month | $55 - $90 / month (amortized) | -87% at >40 Gbps |
| Time to reroute after fiber cut | 90 - 300 sec (BGP reconvergence) | <50 ms (FRR / TI-LFA) | sub-second vs minutes |
| Provisioned capacity headroom | Shared, congested at peak | Dedicated, 40% buffer typical | Predictable performance |
From Transit Contracts to Lit Fiber: A Practical Rollout Playbook
Don't try to build a nationwide backbone on day one — that's how you burn $20M before you’ve moved a single gigabyte off transit. The smartest CDNs we've seen start with their top 3-5 traffic corridors, which typically account for 65-75% of all inter-PoP traffic. If you're pushing 40 Tbps globally, chances are 25+ Tbps is just shuffling between Ashburn, Frankfurt, Singapore, and Los Angeles. Light up those routes first with long-term leases or IRUs, prove the economics, then expand. We usually see the tipping point around 8-10 Tbps of sustained backbone traffic, where owning a 100G wave at $8k-12k/month beats paying $0.20-$0.40/Mbps for transit on the same path.
On the hardware side, you’ve got two real choices: lease dark fiber and light it yourself with coherent DWDM, or lease lit waves (100G, 400G ZR+). Lighting your own fiber gives you 44x 100G channels per pair with modern flex-grid gear, but you’ll need a team that actually understands optical budgets. Most teams starting out take the middle road — leased 100G/400G waves from carriers like Zayo or Lumen, and their own IP/MPLS layer on top with Juniper MX10008s or Arista 7500R3s. This is where Software-Defined Networking for CDN: Programmable Traffic Control becomes non-negotiable. You can't manually manage traffic engineering when you've got 500+ edge PoPs and backbone links flapping.
The migration itself is where people get hurt. Don't cold-cut your transit. We run a 6-8 week shadow period where the private backbone carries 5%, then 25%, then 50% of eligible traffic via BGP communities and selective prepending, while transit stays as the failover. It lets you validate latency, packet loss, and bufferbloat under real load. And if you’re virtualizing edge functions instead of dropping dedicated appliances everywhere, take a look at Network Function Virtualization in CDN Infrastructure: Virtual Appliances at Scale — it’ll save you a forklift upgrade at every PoP you connect.
# Example: Steering cache-fill traffic onto private backbone via BGP communities
# On backbone edge router - prefer private backbone for origin fetch (AS 65001)
route-map BACKBONE-PREF permit 10
match community 65001:100 # Tag for cache-fill / inter-PoP traffic
set local-preference 300
set community 65001:1000 additive # Signal to SDN controller
router bgp 65000
neighbor 198.51.100.1 remote-as 65001 # Private backbone peer
neighbor 198.51.100.1 send-community
neighbor 203.0.113.1 remote-as 3356 # Transit peer (Lumen)
neighbor 203.0.113.1 route-map TRANSIT-DEPREF in
route-map TRANSIT-DEPREF permit 10
set local-preference 80 # De-preference transit for backbone-eligible prefixesAdvanced Traffic Engineering and Troubleshooting When It Breaks
Once you own the fiber, you finally get to do cold-potato routing properly. Instead of handing traffic to the nearest transit peer (hot-potato) and hoping for the best, you haul it across your own backbone to the egress closest to the eyeball. That’s a 15-35ms win on intercontinental routes — we measured 42ms off Frankfurt-to-Singapore p95 just by keeping traffic on-net until Singapore instead of dumping it in London. Pair that with a centralized SDN controller that sees real-time link utilization and you can do things transit will never let you: shift 400 Gbps of VoD mid-roll off a congested 78% utilized wave in under 2 seconds. Your topology choice matters enormously here, so review CDN Backbone Topology Design: Ring, Mesh, and Hierarchical Architectures before you commit to a ring that’ll haunt you at 3 AM.
And it will break. Fiber cuts aren't an "if," they're a weekly event at scale. A single backhoe in Marseille once took out two supposedly diverse paths that shared a bridge conduit — we lost 1.2 Tbps for 90 seconds. Without BFD and fast-reroute, BGP convergence alone will leave you blackholing for 90-120 seconds, which is an eternity for live video. Tune BFD to 300ms x 3, enable TI-LFA or MPLS FRR, and aim for sub-50ms failover. The other silent killer is congestion-induced bufferbloat on your own backbone. If you run links consistently above 65% sustained, tail latency explodes — p99 jumps from 45ms to 180ms+. We've found keeping a 35-40% headroom buffer and auto-throttling low-priority cache-fill traffic is cheaper than over-provisioning everything to 400G.
The last trap is operational complexity. A transit-heavy CDN needs maybe 2-3 senior network engineers. A private backbone CDN needs NOC coverage, optical expertise, and someone who actually understands IS-IS at scale. If your team is under 5 strong netengs, don't build. Also, audit your costs quarterly. We've seen teams celebrate saving $2M/month on transit while quietly spending $1.7M on wave leases, cross-connects, and router maintenance contracts they forgot to amortize. The win is real, but only if you measure the full stack — including the 3 AM pages.
Frequently Asked Questions
At what traffic volume does a private backbone actually become cheaper than transit?
There's no single number, but the math usually flips between 5-10 Tbps of sustained inter-PoP traffic. If you're paying $0.25/Mbps for transit and a 100G wave costs you $10k/month on a key route, that wave pays for itself at around 30-35 Gbps of committed traffic on that route alone. Model your top 5 routes — if 3 of them cross that threshold, it's time to start building.
Should we use dark fiber or leased waves for our first backbone?
Start with leased waves (100G/400G). They're 60-70% cheaper to operationalize, you can light them in 30-45 days instead of 6 months, and you don't need an optical engineering team. Move to dark fiber + coherent DWDM only when you need 10+ waves on a single route or want sub-5ms control over optical restoration. Most CDNs don't need to own glass until they're well past 50 Tbps.
How do we keep latency low when we own the long-haul path?
Own the path, then engineer it. Use cold-potato routing to keep traffic on-net, tune your IGP metrics to prefer low-latency fiber paths (not just shortest hop count), and deploy an SDN controller for real-time TE. We also recommend testing with real user measurements (RUM) — synthetic pings lie. A well-tuned private backbone should beat transit p95 latency by 20-30% on routes longer than 2,000km.
What’s the biggest mistake teams make when transitioning off transit?
Trying to move 100% of traffic at once and underestimating diversity. Never put transit and backbone in the same conduit, and never decommission transit completely — keep at least 20-30% transit as backup and for regions you don't serve on-net. The best architectures treat transit as your insurance policy, not your competitor. You'll still need it for long-tail eyeball networks and DDoS scrubbing.