Back in 2019 we were chasing a throughput ceiling that didn’t make sense. Our edge caches in Ashburn had 100G NICs, plenty of PCIe bandwidth, and yet we couldn’t push more than ~65 Gbps between origin shields and edge without the CPUs melting. We weren’t losing packets — we were drowning in them. Switching that tier to jumbo frames didn’t magically fix everything overnight, but it cut our packet rate by a factor of six and gave us back the headroom we’d been looking for.
Why 1500 Bytes Stops Making Sense Inside the Data Center
On the public internet, 1500 bytes is the lowest common denominator you have to live with. Inside your own data center fabric, you control every switch, NIC, and hypervisor in the path, so that constraint disappears. When you’re moving large objects — video segments, container images, full-site cache fills — between CDN nodes, you’re sending the same payload but chopping it into six times as many frames as you need to. Each of those frames carries Ethernet, IP, and TCP overhead, requires a header lookup, and triggers interrupt handling on the receiving host.
I’ve seen teams spend weeks tuning kernel parameters and RSS queues trying to squeeze another 10% of PPS capacity, when the simpler answer was to send fewer, larger packets. At 100 Gbps with 1500-byte MTU, you’re looking at roughly 8.1 million packets per second in one direction. That’s a brutal rate for even modern CPUs to handle, especially if you’ve got iptables, conntrack, or XDP programs in the path. With MTU 9000, that same 100 Gbps stream drops to about 1.35 Mpps. You haven’t changed your throughput, but you’ve drastically reduced the work your hosts and switches have to do per gigabit.
That doesn’t mean you just flip a switch everywhere. Path MTU Discovery has to be watertight, and any device that can’t handle 9000 bytes will silently blackhole traffic. In our first rollout, we missed a legacy monitoring tap that still had a 1518-byte limit, and it took two days to figure out why cache-fill health checks were flapping. If you’re running an overlay or MPLS in CDN Backbone Networks: Label Switching for Predictable Performance between PoPs, remember that your effective MTU shrinks with every label pushed or VXLAN header added. We standardize on 9000 on the underlay and advertise 8970 to hosts to leave room.
# Verify end-to-end MTU before enabling jumbo frames
ping -M do -s 8972 cache-shield02.ams.internal
# Persistent MTU config on Linux (systemd-networkd)
# /etc/systemd/network/10-bone.network
[Match]
Name=ens1f0
[Link]
MTUBytes=9000
# Verify negotiated MTU and packet errors
ip link show dev ens1f0 | grep mtu
ethtool -S ens1f0 | grep -E 'rx_errors|rx_length_errors|tx_errors'
Where Jumbo Frames Actually Help Throughput (And Where They Don't)
Not all CDN traffic benefits equally. If your workload is mostly small HTTP requests, TLS handshakes, and sub-10KB API responses to eyeballs, jumbo frames won’t move the needle — you’re already bottlenecked on round-trips and connection setup, not bytes per packet. The win comes from east-west bulk transfers: shield-to-edge fills, tiered cache rebalancing, log shipping to object storage, and backup replication. Those flows are long-lived, high-throughput, and they stay inside your ECMP Routing in CDN Networks: Equal-Cost Multi-Path Traffic Distribution fabric where you own the entire path.
The other place I’ve measured a clear gain is on hosts doing software encapsulation. If you’re running VPP, OVS, or any userspace datapath that adds VXLAN or GRE, your per-packet CPU cost is even higher. Larger packets amortize that cost. We saw a 30-40% reduction in CPU cycles per gigabyte on our shield nodes after moving to 9000, which let us run fewer shield instances for the same fill throughput. For background, the principles behind modern stack efficiency are well covered in the HTTP Working Group Specifications and practical tuning guides on web.dev Articles, but the kernel’s GRO/GSO behavior is worth understanding directly — Linux will already coalesce incoming 1500-byte frames into larger GRO super-packets, so jumbo frames on the wire complement what the stack is trying to do.
Don’t expect lower latency from jumbos, though. Serialization delay actually goes up slightly with larger frames — a 9000-byte frame takes ~0.72 microseconds on 100G versus ~0.12 microseconds for 1500. For bulk throughput that’s irrelevant, but if you mix latency-sensitive traffic on the same queues, you can introduce jitter. We isolate our cache-fill VLANs and apply separate QoS, and for path selection across our backbone we depend on Segment Routing for CDN Traffic Engineering: Source-Routed Path Selection to keep bulk paths from interfering with interactive traffic. If you can’t isolate, you’ll want to leave your front-end eyeball-facing interfaces at 1500 and only enable 9000 on the backplane.
# Tune host stack for 9000 MTU - reduce interrupts, enable offloads
ethtool -G ens1f0 rx 4096 tx 4096
ethtool -C ens1f0 adaptive-rx on adaptive-tx on rx-usecs 100
ethtool -K ens1f0 gro on gso on tso on lro off
# iperf3 validation between two cache nodes
# Server: iperf3 -s -p 5201
# Client: iperf3 -c 10.20.30.12 -P 8 -t 60 -M 8960 -w 4M
# Check for PMTUD/fragmentation on the path
tracepath -n 10.20.30.12
sysctl net.ipv4.tcp_mtu_probing=1
Measured Performance: MTU 1500 vs. MTU 9000 on 100G Cache Links
We benchmarked this on identical Dell R750s with Mellanox ConnectX-6 Dx 100G NICs, directly connected through a single Arista 7280R3 leaf — no ECMP, no oversubscription, so the numbers are as clean as they get. The test was a simple cache-fill simulation: eight parallel TCP streams pulling 100GB of 1MB objects from NVMe-backed Nginx on the origin shield to a cold edge cache. We ran the same test at both MTUs, three times each, and averaged the results. The host tuning in the snippet above was applied for both cases, so the only variable was frame size.
What stood out wasn’t just the peak throughput — it was the CPU cost to get there. At 1500 bytes, we were pinned on softirq and couldn’t sustain line rate without dropping retransmits. At 9000, we hit ~96 Gbps with 30% idle CPU remaining. That headroom matters more than the raw number, because in production you’re never running a single flow in isolation. You’ve got health checks, telemetry scrapes, and control plane traffic all fighting for cycles on the same host.
If you’re considering a rollout, run this exact test in your own environment before you touch production. Your NIC firmware, switch buffer settings, and even cable length can shift the results. We gate our rollout on three checks: zero rx_length_errors after 24 hours of burn-in, PMTUD working end-to-end for both TCP and UDP, and no increase in tail latency on co-located services.
| Metric (8-flow aggregate) | MTU 1500 | MTU 9000 | Change |
|---|---|---|---|
| Sustained Throughput | 68.4 Gbps | 96.2 Gbps | +40.6% |
| Packets Per Second (avg) | 5.68 Mpps | 1.32 Mpps | -76.8% |
| Avg. Host CPU (shield + edge) | 88.7% | 54.3% | -38.8% |
| TCP Retransmits / 60s | 1,842 | 112 | -93.9% |
| Interrupt Rate (interrupts/sec) | 182k | 41k | -77.5% |
Practical Implementation: Enabling MTU 9000 End-to-End Without Black-Holing Traffic
Flipping MTU to 9000 on a single cache node won't buy you anything — it'll just create a Path MTU Discovery black hole. Jumbo frames only work when the entire Layer-2 domain is consistent: physical NIC, hypervisor vSwitch or SmartNIC, Top-of-Rack, spine, and the storage or origin-shield peer on the other end. Miss one hop and you'll see exactly what we saw in our first rollout: 8972-byte pings succeeding rack-local but silently dropping across a spine that was still at 1500, with TCP retransmits spiking 12x because PMTUD ICMP messages were being filtered by a firewall policy. If you're running virtualized caches, that Network Function Virtualization in CDN Infrastructure: Virtual Appliances at Scale layer is often the culprit — the vSwitch port group or DPDK poll-mode driver needs its MTU bumped independently of the host NIC.
Don't do a flag day. The reliable sequence we've used across 40+ PoPs is: 1) Lab validate with iperf3 and real cache fill workloads, not just synthetic pings, 2) Canary a single rack/leaf pair, and 3) Expand fabric-wide. On Linux, set the MTU persistently (netplan, systemd-networkd, or ifcfg) and verify L2, not just L3. A standard 100G Mellanox/NVIDIA CX6 will show mtu 9000 in ip link but still be capped at 1500 on the switch's LAG. We enforce mtu 9214 on Arista/Juniper to leave headroom for VXLAN or MPLS tags, and we explicitly allow ICMP Type 3 Code 4 (Fragmentation Needed) through every ACL. For segmented fabrics, keep your client/frontend network at 1500 and only enable 9000 on your east-west cache-fill, storage, and backbone replication VLANs.
# --- Linux cache / origin shield node (Ubuntu/Debian) ---
# Set MTU persistently - only on the back-end fabric, not the frontend
sudo ip link set dev eth1 mtu 9000
sudo ip link set dev eth2 mtu 9000
# Verify L2 end-to-end BEFORE pushing to production
# 8972 = 9000 - 20 (IP) - 8 (ICMP) | -M do prohibits fragmentation
ping -M do -s 8972 -c 100 10.50.12.44
tracepath -n 10.50.12.44
# Check for silent drops / reassembly errors
ethtool -S eth1 | grep -E 'rx_errors|rx_length_errors'
! --- Arista EOS (ToR / Spine) - don't forget the port-channel ---
interface Ethernet1
mtu 9214
interface Ethernet2
mtu 9214
interface Port-Channel10
mtu 9214
port-channel lacp fallback
!And clamp your MSS where you have to talk to mixed-MTU peers. For any traffic that might egress the PoP, we set iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu on the border. It costs you nothing and prevents that ugly 30-second TCP hang when a 9000-byte host tries to talk to a 1500-byte origin through a firewall that drops ICMP. Monitor netstat -s | grep retrans and your ToR's giants counters — if they're climbing post-change, you've got a mismatch.
Advanced Tuning, Overlays, and Troubleshooting at Scale
Once the fabric is at 9000, your next bottleneck becomes the overlay. If you're running VXLAN for tenant isolation or Software-Defined Networking for CDN: Programmable Traffic Control, remember the encapsulation tax: VXLAN adds 50 bytes, Geneve can add 58-100+. That means a 9000-byte guest MTU will actually need 9050-9100 on the underlay. We've standardized on 8950 MTU inside VMs/containers and 9214 on the physical underlay to avoid inner fragmentation. In a CDN Backbone Topology Design: Ring, Mesh, and Hierarchical Architectures with MPLS, that tag stack is another 4-16 bytes per label. Get this wrong and you'll see throughput look great on iperf but collapse on real objects because every other packet is being fragmented in software by the kernel.
You'll also want to retune your offloads. Jumbo frames make GRO/LRO and TSO dramatically more effective — we typically see interrupt coalescing drop from ~8.1 Mpps to ~1.35 Mpps at 100 Gbps, which translates to a 28-34% reduction in softirq CPU on NGINX/ATS cache nodes. But don't leave tcp_rmem and tcp_wmem at their defaults; with 9000-byte segments you'll want to bump net.core.rmem_max to 16MB and ensure your congestion window can actually fill the pipe. Conversely, if you're seeing microburst-induced drops, check your switch buffers. A single 9000-byte frame occupies 6x the buffer of a 1500-byte one, so on shallow-buffered ToRs (like Trident3) you can actually increase tail latency by 150-300µs during incast. We've had to enable ECN and tune WRED thresholds separately for jumbo-enabled queues.
When it breaks, it's almost never obvious. If throughput stalls but pings work, try ping -M do -s 8972 AND ping -M do -s 1472 hop-by-hop — a success on small and failure on large points directly to the misconfigured interface. Check netstat -s | grep "PMTU" and look for "frag needed" drops in your firewall logs. And disable NIC scatter-gather bugs: older ConnectX-4 firmware would silently truncate 9000-byte frames to 1500 when rx-gro-hw was on. We caught it because cache fill throughput between two racks was stuck at ~42 Gbps on a 100G link, while intra-rack was line-rate. A firmware bump fixed it instantly.
Frequently Asked Questions
Should I enable jumbo frames on internet-facing or edge client connections?
No, and you shouldn't try. The public internet is still overwhelmingly 1500 bytes — we see less than 0.3% of client paths supporting over 1500 via PMTUD. Enable jumbo frames only inside the data center: cache-to-cache fill, cache-to-storage (NVMe-oF, Ceph), and PoP-to-PoP backbone replication. Keep your frontend/eyeball-facing NICs and load balancers at 1500 to avoid PMTUD black holes and excessive retransmits for end users.
What happens if one switch or NIC in the path doesn't support MTU 9000?
You'll get fragmentation or drops, and it's brutal for throughput. If PMTUD is working and ICMP isn't filtered, TCP will eventually discover the 1500 limit and fall back, but you'll add 200-500ms of initial connection stalls. If ICMP is filtered (common), you get a black hole: large packets are dropped silently, TCP hangs until timeout, and throughput collapses by 70-90%. That's why we require end-to-end verification with ping -M do and enforce a consistent 9214 MTU on every L2 device in the fabric.
Do jumbo frames increase latency or cause bufferbloat?
Per-packet serialization latency goes up slightly — a 9000-byte frame takes ~0.72µs on 100G vs. ~0.12µs for 1500, so you're adding about 600ns on the wire. But you'll send 6x fewer packets, so interrupt and processing latency usually drops. The risk is switch buffer occupancy during incast. With 9000-byte frames, a 32MB shared buffer holds ~3,500 frames instead of ~21,000, so incast from 10 caches can cause tail latency spikes of 0.2-0.8ms if you don't have ECN or proper QoS. Tune buffers, don't just enable jumbos and walk away.
Is 9000 the right MTU, or should I use 4000 or 8000?
For most CDN workloads on modern hardware, 9000 (or 8950 to leave room for overlays) is the sweet spot. We tested 4000 vs. 9000 on 100G cache fill: 4000 gave us ~60% of the PPS/CPU reduction, while 9000 gave us the full 6x reduction and an extra 4-5% throughput gain on 50MB+ objects. There's no real downside to 9000 vs. 4000 on switches that support 9216+, so we standardize on 9000. Just avoid exotic values like 9600 — some NICs/drivers round down or fail to negotiate cleanly.