CDN Large File Download Optimization: Multi-GB Delivery Guide

Large file delivery — software updates, game patches, ISO images, firmware distributions, scientific datasets, and container images — represents a fundamentally different CDN workload than web page assets or streaming media. A single 20 GB game patch downloaded by 10 million players generates 200 petabytes of transfer in days. The files are large enough that individual downloads last minutes to hours, connections must survive network interruptions, and the thundering herd pattern (millions of clients requesting the same file simultaneously) can overwhelm CDN infrastructure designed for smaller objects.

This guide examines the CDN engineering techniques that make reliable, high-throughput large file delivery possible: cache slicing, byte-range resume support, connection tuning, prefetch strategies, and the infrastructure architecture that handles launch-day traffic spikes for major software releases.

Why Large Files Are Different

Standard CDN caching assumes objects are small enough to fit in edge cache storage and fast enough to transfer that connection lifetime is measured in milliseconds. Large files violate both assumptions. A 15 GB file at 50 Mbps takes 40 minutes to download. During those 40 minutes, the TCP connection must remain alive, survive network blips, maintain congestion window state, and compete with thousands of other concurrent long-lived connections on the same edge server.

Three characteristics distinguish large file delivery from standard CDN traffic:

Cache economics change. A single 20 GB file occupies the same cache space as 20,000 typical web page assets. Edge PoPs with 1–2 TB of cache storage can only hold 50–100 large files before eviction pressure forces out other content. Cache slicing (described below) transforms this dynamic by allowing partial caching and independent chunk management.

Connection patterns change. Web traffic opens and closes connections in seconds. Large file downloads hold connections for minutes to hours. An edge server handling 10,000 concurrent large file downloads must maintain 10,000 long-lived TCP connections simultaneously, each with its own congestion window, send buffer, and socket state. This stresses different system resources (memory for socket buffers, file descriptors) than high-request-rate web traffic.

Failure modes change. When a 5 KB JavaScript file fails to transfer, the client retries the entire request — the cost is negligible. When a 15 GB download fails at 14.5 GB (97% complete), restarting from zero wastes hours of client time and CDN bandwidth. Resume support via byte-range requests is not optional for large file delivery; it is a core functional requirement.

Cache Slicing Architecture

Cache slicing is the foundational technique for large file CDN delivery. Instead of caching a 15 GB file as a single object, the CDN divides it into fixed-size slices (typically 1–8 MB) and caches each slice independently. This solves multiple problems simultaneously.

Time-to-first-byte on cache miss drops from minutes (waiting for the CDN to download the entire file from origin) to milliseconds (waiting for only the first slice). The CDN begins serving the client as soon as the first slice arrives while continuing to fetch subsequent slices from origin in parallel.

# Nginx cache slicing configuration for large files
location /downloads/ {
    # Slice large files into 4 MB chunks
    slice 4m;
    proxy_set_header Range $slice_range;

    proxy_pass http://origin_backend;
    proxy_cache large_file_cache;
    proxy_cache_valid 200 206 7d;
    proxy_cache_key $uri$slice_range;

    # Support client byte-range requests
    add_header Accept-Ranges bytes;
    add_header Cache-Control "public, max-age=604800";

    # Prevent cache lock storms: serve stale while updating
    proxy_cache_use_stale updating;
    proxy_cache_lock on;
    proxy_cache_lock_timeout 10s;

    # Limit origin bandwidth per slice fetch
    proxy_limit_rate 50m;
}

The proxy_cache_lock directive prevents thundering herd on individual slices. Without it, 1,000 concurrent clients requesting the same uncached file trigger 1,000 parallel origin requests for slice 1, then 1,000 for slice 2, and so on. With cache locking, only the first request for each slice triggers an origin fetch; subsequent requests wait for the cache to fill and then serve from cache. This reduces origin load from clients × slices to just slices requests per file per edge PoP.

Download Resume and Integrity

Reliable resume support requires correct implementation of three HTTP mechanisms: byte-range requests, entity tags (ETags), and conditional requests. Together, these ensure that a resumed download continues from exactly where it stopped, with the same file version.

When a client resumes a download, it sends two headers: Range: bytes=X- (requesting bytes from position X onward) and If-Range: "etag-value" (confirming the file hasn't changed). If the ETag matches, the CDN responds with 206 Partial Content and the remaining bytes. If the ETag doesn't match (the file was updated), the CDN responds with 200 OK and the complete new file, signaling the client to restart the download.

# Resume request flow
# Initial download (interrupted at byte 5368709120)
GET /patches/game-v2.1.pak HTTP/1.1
Host: cdn.example.com
→ 200 OK
   Content-Length: 15032385536
   ETag: "a7f3b2c1d4e5"
   Accept-Ranges: bytes
   [... 5368709120 bytes transferred, connection lost ...]

# Resume request
GET /patches/game-v2.1.pak HTTP/1.1
Host: cdn.example.com
Range: bytes=5368709120-
If-Range: "a7f3b2c1d4e5"
→ 206 Partial Content
   Content-Range: bytes 5368709120-15032385535/15032385536
   Content-Length: 9663676416
   [... remaining 9.6 GB ...]

For CDN operators, ETag consistency across edge servers is critical. If different edge PoPs generate different ETags for the same file (which happens when ETags are based on inode numbers or server-specific attributes), a client that starts downloading from one PoP and resumes from another will fail the If-Range check and restart from zero. ETags should be based on content-derived values (file hash, content length + modification time) that are identical regardless of which server generates them.

Connection Tuning for Large Transfers

TCP connections carrying large file downloads require different kernel and application tuning than connections serving web pages. The core difference is that large downloads are bandwidth-delay product (BDP) limited: the maximum throughput equals the TCP window size divided by the round-trip time. For a 100 Mbps connection with 50ms RTT, the BDP is 625 KB — the TCP window must reach at least 625 KB to fully utilize the connection.

# Linux kernel tuning for large file CDN delivery

# TCP buffer sizing: allow large windows for high-BDP connections
net.core.rmem_max = 16777216      # 16 MB max receive buffer
net.core.wmem_max = 16777216      # 16 MB max send buffer
net.ipv4.tcp_rmem = 4096 1048576 16777216  # min/default/max
net.ipv4.tcp_wmem = 4096 1048576 16777216

# Congestion control: BBR provides better throughput for long transfers
net.ipv4.tcp_congestion_control = bbr
net.core.default_qdisc = fq

# Connection handling for long-lived downloads
net.ipv4.tcp_keepalive_time = 300    # Detect dead connections
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_slow_start_after_idle = 0  # Don't reset cwnd on idle

# File descriptor limits for concurrent connections
fs.file-max = 1048576
net.core.somaxconn = 65535

BBR congestion control (described in detail in the BBR congestion control guide) is particularly effective for large file delivery because it probes available bandwidth rather than relying on packet loss as a congestion signal. On paths with bufferbloat (common on residential connections), BBR maintains higher throughput than CUBIC by avoiding unnecessary window inflation that fills intermediate buffers.

The tcp_slow_start_after_idle setting is especially important for large downloads. Standard TCP resets the congestion window to its initial value after a connection has been idle for one RTO (retransmission timeout). During a large download, brief pauses (client processing, disk I/O stalls) can trigger this reset, causing throughput to temporarily collapse. Disabling slow start after idle maintains the established congestion window across brief pauses.

Thundering Herd Mitigation

The most challenging aspect of large file CDN delivery is the thundering herd pattern during major releases. When a game studio releases a 20 GB patch, millions of clients begin downloading within minutes. Without mitigation, this creates cascading failures: every edge PoP simultaneously requests the full file from origin, origin bandwidth saturates, failed cache fills trigger retries, and the retry storm amplifies the overload.

Mitigation operates at three layers. At the origin layer, an origin shield absorbs the fan-out. Instead of 200 edge PoPs each requesting the file from origin, all edge PoPs request from a shield layer that makes a single origin request and distributes to edges. The shield should have dedicated high-bandwidth connectivity to the origin (10–100 Gbps) and large cache storage to absorb the initial burst. For more on shield patterns, see the cache hierarchy guide.

At the edge layer, cache locking (one origin request per slice per PoP) and request coalescing (multiple client requests for the same slice share a single origin fetch) prevent amplification. Stale-while-revalidate for file metadata allows the edge to begin serving before freshness checks complete.

At the client layer, staggered rollout distributes demand over time. Rather than making the update available to all clients simultaneously, a phased rollout releases to 1% of clients initially, monitors CDN performance, then expands to 10%, 50%, and 100% over several hours. Client-side jitter (each client waits a random 0–300 second delay before starting the download) further smooths the demand curve. The publisher controls these parameters, but the CDN must communicate capacity headroom to inform rollout decisions.

Delta Updates and Differential Delivery

The most effective optimization for large file delivery is reducing the file size itself. Delta updates (also called differential or patch updates) compute the binary difference between the old version and the new version and deliver only the changed bytes. A 20 GB game that changes 500 MB of content between versions produces a 500 MB–2 GB delta rather than a 20 GB full download.

CDN operators support delta delivery by hosting multiple delta files (one per source version) alongside the full file. The client reports its current version, and the CDN routes to the appropriate delta. If the delta is unavailable or the client's version is too old for incremental updating, it falls back to the full download. This requires CDN edge logic (typically an edge worker or custom routing rule) that inspects the client request and selects the correct file variant.

Container image distribution uses a similar approach: OCI image layers are content-addressed blobs that are shared between image versions. Pulling a new version of an image only downloads the layers that changed. CDN caching of individual layers (identified by their SHA256 digest) means popular base layers (like ubuntu:22.04) achieve very high cache hit ratios across the entire container ecosystem, regardless of which image includes them.

Integrity Verification

Large file downloads are more susceptible to corruption than small transfers because the probability of a bit error scales with transfer size, and the cost of re-downloading on corruption is high. CDN infrastructure supports integrity verification at multiple levels.

Transport-level integrity via TLS ensures data is not corrupted or tampered with during transit. All modern CDN delivery uses HTTPS, providing both encryption and integrity. However, TLS protects only the link between client and edge server — it does not detect corruption in CDN cache storage or at the origin.

Application-level integrity via content hashes (SHA256, SHA512) allows the client to verify the complete downloaded file against a known hash published alongside the download link. The publisher computes and publishes the hash; the client verifies after download. CDN infrastructure facilitates this by serving the hash file (typically SHA256SUMS) alongside the download, cached with appropriate TTLs.

Chunk-level integrity via hash trees (Merkle trees) enables verification of individual chunks without downloading the complete file. Each chunk has its own hash, and the tree of hashes is signed by the publisher. The client verifies each chunk as it arrives and immediately detects corruption without waiting for the complete download. This approach, used by protocols like BitTorrent and IPFS, is increasingly adopted by software update systems. For more on content integrity in CDN delivery, see the supply chain security guide.

Measuring Large File Delivery Performance

Large file CDN performance metrics differ from web content metrics. Page load time and time-to-interactive are irrelevant; what matters is download throughput (measured across the full transfer, not just the first few seconds), completion rate (percentage of started downloads that finish successfully), resume success rate (percentage of interrupted downloads that successfully resume), and time-to-completion (total wall-clock time including interruptions and resumes).

Monitoring should track these metrics segmented by file size, geographic region, ISP, and connection type (fiber, cable, cellular). Large file delivery performance varies dramatically across these dimensions — a user on fiber in Seoul has a fundamentally different experience from a user on satellite in rural Australia, and the CDN must optimize for both.

Edge server health metrics specific to large file delivery include concurrent long-lived connections (should not exceed server capacity), cache fill ratio (percentage of slices served from cache vs origin), origin bandwidth utilization during launch events, and TCP retransmission rate per connection (indicates last-mile congestion). These metrics feed capacity planning models that predict how many concurrent large file downloads each edge PoP can sustain at target throughput levels. For a broader monitoring framework, see the CDN observability guide.