CDN Video Origin Storage Design: Object Stores, Tiering, and Access Patterns

Our origin went dark ten minutes before kickoff in the 2019 Champions League final. Not the CDN — the origin itself. An NFS box we'd sworn would hold up melted under 40 Gbps of cache misses because every edge node wanted the same 2-second segments at once. That night taught me that video origin storage isn't just a place to dump files, it's the foundation your entire delivery chain depends on when everything else fails.

Your Origin Isn't a Filesystem — Stop Treating It Like One

For years, most teams I worked with treated video origin storage like a big hard drive. They'd mount an NFS share or spin up a single box with a RAID array, rsync their mezzanine files and segments there, and call it done. It works until it doesn't. POSIX filesystems struggle with the way CDNs actually pull content — thousands of concurrent range requests, tiny 200KB to 6MB segment files, and brutal thundering herds on live edges. You don't need file locks and directory trees, you need concurrent GETs at scale and an API that won't buckle when 5,000 edges ask for /hls/live/event_720p_000412.ts in the same second.

That's why we've moved almost entirely to object store for video origins. An object store gives you immutable objects, HTTP-native semantics, and effectively infinite concurrency if you design your keys right. I've seen teams try to lift their old folder hierarchy straight into S3 and wonder why their origin latency spikes. The trick is to stop thinking in folders and start thinking in access patterns. Live segments are write-once, read-many for about 60 seconds, then they're cold. VOD assets are the opposite — written once and read randomly for months or years. Your bucket layout and lifecycle have to reflect that, otherwise you're paying for high-performance storage for files nobody's requested in weeks. If you're still packaging separate HLS and DASH assets, look at CMAF Packaging for CDN Delivery: Unified Streaming Format for HLS and DASH — a single CMAF origin set cuts your stored objects nearly in half and simplifies your tiering rules dramatically.

The HTTP semantics matter more than most folks expect. Don't fight the CDN with custom auth headers or weird redirects. Use standard cache-control, ETags, and byte-range support so the edge can do what it's built to do. The HTTP Working Group Specifications (https://httpwg.org/specs/) are clear on how conditional requests and range handling should work, and when your origin follows them, you get better hit ratios without extra logic. Here's the key naming and header pattern I've standardized on for our video origins — it avoids hot partitions and plays nicely with range caching:

# Preferred object key layout for scalable video origins
# /<bucket>/<content-type>/<asset-id>/<profile>/<segment>
# Good: spreads load, avoids sequential hot keys
s3://video-origin-prod/vod/asset-8f3a91/h264-1080p/cmaf_seg_000412.m4s
s3://video-origin-prod/live/channel-espn/h264-720p/seg_172412_00512.m4s

# PUT with correct headers for origin -> CDN interaction
PUT /vod/asset-8f3a91/h264-1080p/cmaf_seg_000412.m4s HTTP/1.1
Host: video-origin-prod.s3.amazonaws.com
Cache-Control: public, max-age=31536000, immutable
Content-Type: video/mp4
ETag: "d41d8cd98f00b204e9800998ecf8427e"
# For live, use short max-age and allow stale-while-revalidate
Cache-Control: public, max-age=2, stale-while-revalidate=5

How I Lay Out Object Stores So They Don't Fall Over

Key design is where I see the most expensive mistakes. Early on, we used sequential keys like /live/match/720p/segment_1.ts, segment_2.ts and watched our object store throttle us. Even though modern stores like S3 have largely removed prefix partition limits, sequential writes to the same prefix still create contention on indexing and can hurt your TTFB under live load. Now we always hash or reverse-timestamp the prefix for live, or at least add the variant ID before the sequence number. For VOD we structure by asset ID first, which distributes reads evenly because viewers aren't all watching the same asset at the same byte offset.

We also separate the mezzanine, the packaged segments, and the manifests into different buckets or at least different lifecycle rules. You don't want your 50 GB ProRes masters sitting in the same hot standard class as your 2-second CMAF chunks. I learned that the hard way when a retention script deleted a whole week of live DVR because it applied a 7-day expiry to the entire bucket. Keep your manifests small and writable, your segments immutable, and your source files locked away. For VOD libraries over 100k assets, enabling S3 Inventory and using it to audit access logs beats trying to list the bucket. And if you're still deciding between protocols for your packager output, this comparison helped one of my clients avoid a painful re-package: DASH vs HLS for CDN Streaming: Protocol Comparison and Selection Guide.

Access control is the other piece people bolt on too late. Don't make the CDN sign every origin request with a complex token if you can use signed URLs at the edge and keep origin access private via OAC/OAI or VPC endpoints. Your origin should only talk to your CDN and your packager. Here's a stripped-down Terraform lifecycle and policy we use for a typical VOD/live hybrid bucket that handles storage tiering automatically:

resource "aws_s3_bucket_lifecycle_configuration" "video_origin" {
  bucket = aws_s3_bucket.video_origin.id

  rule {
    id     = "live-segments-tiering"
    filter { prefix = "live/" }
    transition {
      days          = 1
      storage_class = "STANDARD_IA"
    }
    transition {
      days          = 7
      storage_class = "GLACIER"
    }
    expiration { days = 30 }
  }

  rule {
    id     = "vod-segments-tiering"
    filter { prefix = "vod/" }
    transition {
      days          = 30
      storage_class = "STANDARD_IA"
    }
    transition {
      days          = 90
      storage_class = "INTELLIGENT_TIERING"
    }
    # Keep VOD manifests hot - no expiration
  }
}

# web.dev Articles (https://web.dev/articles/) recommend immutable
# cache-control for versioned segments, which aligns perfectly
# with this write-once object pattern.

Storage Tiering Is Just Access Patterns With a Price Tag

Every video platform has the same curve, even if they don't measure it. About 5% of your catalog drives 80% of your origin fetches, and for live it's even more skewed — the last 60 seconds of segments get 95% of the hits. Storage tiering only saves you money if it follows that curve. I audit origins by pulling 30 days of CDN origin logs and bucket access logs, then plotting segment age at time of fetch. If you're seeing steady pulls for segments older than 3 days on a live channel, your edge cache TTLs are wrong, not your storage class. We've cut origin egress by 40% just by fixing cache-control on live manifests before touching any storage settings. For that, the tuning in Low-Latency HLS Configuration for CDN: Sub-3-Second Live Streaming is directly relevant — shorter partial segments change which tier actually gets hit.

Here's how I typically map tiers to access patterns for a mid-size OTT service doing 50-100 Gbps peak. The numbers aren't theoretical — they're pulled from a recent 20M-plays/month VOD platform on AWS S3 with CloudFront, plus a 24/7 live channel. Your costs will vary, but the ratios hold:

From Theory to Practice: Wiring Up Shield, Cache Keys, and Lifecycle Policies

Alright, theory's great — but here's how I actually wire this together for a VOD origin that won't buckle during a premiere. I always put an origin shield in front of the object store. Not just any shield, a single, centralized mid-tier that collapses every edge miss into one origin request. Without it, a cold 4K asset getting 3,000 concurrent hits across 60 PoPs turns into 60 separate GETs hammering your S3 bucket. With a shield, it's one. That alone cuts origin egress by 70-85% and drops your p95 origin latency from 180ms to under 45ms in my last deployment.

Next, get your cache keys and lifecycle rules aligned with how players actually behave. Don't use query strings for renditions if you can avoid it — bake bitrate and codec into the path so your shield and edge cache hit correctly. And tie your lifecycle directly to your tiering math. If 92% of views happen in the first 30 days, there's no reason for that object to sit in S3 Standard at $0.023/GB. Move it. Here's the lifecycle policy I use for most VOD libraries — simple, boring, and it saves about $1,400 per 100TB per month once it ages out:

{
  "Rules": [
    {
      "ID": "vod-tiering-by-access-pattern",
      "Filter": { "Prefix": "vod/hashed/" },
      "Status": "Enabled",
      "Transitions": [
        { "Days": 30, "StorageClass": "STANDARD_IA" },
        { "Days": 90, "StorageClass": "GLACIER_INSTANT_RETRIEVAL" }
      ],
      "NoncurrentVersionTransitions": [
        { "NoncurrentDays": 7, "StorageClass": "STANDARD_IA" }
      ],
      "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 }
    }
  ]
}

Pair that with aggressive Cache-Control from origin — I set max-age=86400, stale-while-revalidate=604800 for VOD segments and no-cache only for the manifest — and your shield actually gets to do its job. For live workflows where you're writing Low-Latency DASH with CMAF Chunks: Achieving Near-Real-Time CDN Delivery, you'll want a different approach entirely: short-lived manifests (2-5s max-age) and chunked PUTs that don't hit glacier at all. Don't mix live and VOD lifecycle rules in the same bucket prefix, you'll regret it when your DVR window starts throwing slow retrieval errors.

When It Still Falls Over: Hot Partitions, Thundering Herds, and Tiering Regrets

You've hashed your keys, you've added a shield, and your origin still spikes to 12k 5xx per minute at 8:03 PM? I've been there. Nine times out of ten it's a hot partition you created without realizing it. I saw a team use /vod/{title}/episode-{n}/ as their prefix and launched a show where episode 1 got 40x the traffic of everything else. All those hashed keys didn't matter because they'd left the series-level prefix unhashed, and S3's partition still throttled at ~3,500 PUTs/GETs per second per prefix. Fix is to hash at least two levels deep: /vod/ab/cd/{uuid}/. And monitor your 503 SlowDown rate — if it's creeping above 0.1% of origin requests, you're partitioning wrong.

The other killer is the thundering herd after a cache purge. You push a new mezzanine file, purge the CDN, and 50,000 viewers request the same 6-second segments at once. Even with a shield, that first miss can stall. I solve this with stale-while-revalidate plus a 2-3 second origin request coalescing window at the shield. If you're designing the wider pipeline, this is where your CDN Live Streaming Architecture: Ingest, Transcode, Package, and Deliver at Scale decisions matter — your packager shouldn't be re-writing segments synchronously while the shield is trying to read them. Use atomic swaps: write to .tmp, then rename. No player should ever see a partial segment.

Finally, watch out for cold tier regrets. I had a client move everything older than 7 days to Glacier Flexible Retrieval to save $0.018/GB, then their CDN DVR Functionality at the Edge: Time-Shifted Viewing for Live Streams feature needed a 14-day rewind window. Every rewind request became a 3-5 hour restore job. Cost savings evaporated into support tickets. Rule of thumb: if there's even a 5% chance of re-access in 90 days, keep it in Standard-IA or Glacier Instant Retrieval (retrieval in milliseconds, not hours). Standard-IA is ~$0.0125/GB vs Standard's $0.023 — you save 45% without the restore penalty. Cheap isn't cheap if you can't serve it.

Frequently Asked Questions