Origin Storage Backends: S3, GCS, Azure Blob, and Custom Object Stores for CDN

Back in 2018 I got paged at 2 AM because a client's single Nginx origin had filled its disk and stopped serving a 40 GB video library. We'd warned them for months, but they kept treating that one VM like it was infinite. The next morning we migrated that library to S3, pointed the CDN at the bucket, and we haven't worried about disk space since. If you're just getting started with CDN architecture, this shift from server to object store is probably the most important design decision you'll make.

Why Object Storage Replaced the Traditional Origin Server

For years the default CDN origin was just a web server you managed yourself — Apache, Nginx, maybe a small cluster behind a load balancer. It worked, but it meant you were responsible for disks, RAID, backups, and scaling that box every time traffic spiked. I've spent more weekends than I'd care to admit rsyncing files between origins to keep them in sync. Object storage flipped that model. Services like S3, GCS, and Azure Blob give you what feels like an infinite hard drive behind an HTTP API, with 11 nines of durability so you don't have to think about losing a file.

From the CDN's perspective, an object store is just another HTTP origin, but a very well-behaved one. When a user requests /images/hero.jpg and it isn't cached at the edge, the CDN makes a simple GET request to your bucket to pull it. This is called an origin pull, and it's defined by the same HTTP semantics you'll find in the HTTP Working Group Specifications. The bucket doesn't need to run your app logic; it just needs to return the object with the right headers — Cache-Control, ETag, and Content-Type matter a lot more than you'd expect. Getting those headers right at upload time saves you a ton of cache invalidation headaches later.

There are trade-offs you don't see until you've run it in production. Object stores are eventually consistent in some regions and operations, listing can lag, and a 404 from storage doesn't always mean the file is truly gone. I learned that the hard way when a batch delete job raced with CDN prefetches and we served stale 404s for five minutes. That's why you'll want to pair your origin storage with proper Origin Health Monitoring Patterns: Active Probes, Passive Checks, and Hybrid Approaches so your CDN can tell the difference between a missing object and a sick origin.

# Example: Minimal CDN origin configuration for an S3 bucket
# This is conceptually similar for Fastly, CloudFront, or Varnish

origin "s3-origin" {
  hostname = "my-cdn-assets.s3.us-east-1.amazonaws.com"
  protocol = "https"
  port = 443

  # Forward the Host header expected by S3
  host_header = "my-cdn-assets.s3.amazonaws.com"

  # Keep connections warm - critical for performance
  keepalive = true
  keepalive_timeout = 60s

  # Only allow the CDN to fetch via GET/HEAD
  methods = ["GET", "HEAD"]
}

Wiring Up S3, GCS, and Azure Blob as CDN Origins

At a high level they all do the same thing, but the details will trip you up if you're not careful. AWS S3 wants requests in the form bucket.s3.region.amazonaws.com/key and if your bucket is private, your CDN has to sign every origin request with AWS Signature V4. Google Cloud Storage is similar but it offers two flavors: the native storage.googleapis.com endpoint and an S3-compatible XML API, which is handy if you're using tooling built for S3. Azure Blob is the odd one out — its URL structure is account.blob.core.windows.net/container/blob and its authentication uses a completely different shared-key scheme. Don't mix them up; I've burned a whole afternoon debugging 403s because I sent an AWS Authorization header to Azure.

For public read buckets, setup is almost trivial. You mark the bucket or container as public, disable block-public-access as needed, and point your CDN's origin hostname at it. For private buckets, you'll configure your CDN to add authentication. Most modern CDNs can generate signed requests on the fly so you don't have to expose your storage credentials. If you're learning, start with a public test bucket and then lock it down. Articles on web.dev Articles do a good job explaining how Cache-Control from your origin controls edge caching, and that logic applies whether your origin is S3 or a custom store.

You don't have to use the big three, either. A lot of teams I've worked with run S3-compatible stores like MinIO, Ceph RADOS Gateway, or Wasabi because they need data residency or lower egress costs. These custom object stores speak the same S3 API, so your CDN config barely changes — just the hostname and maybe TLS settings. The catch is you're now responsible for durability and scaling again. If you go this route, pay close attention to Origin Connection Pooling Optimization: Reducing TCP Overhead at CDN Scale and Origin Retry Strategies for CDN Resilience: Backoff, Jitter, and Budget Limits. A self-hosted store can't handle thousands of short-lived TLS handshakes as gracefully as S3 can, so connection reuse and smart retries become essential.

# Private S3 origin fetch with on-the-fly request signing
# Example edge logic (VCL-style pseudocode)

sub vcl_backend_fetch {
  set bereq.http.Host = "my-private-assets.s3.us-east-1.amazonaws.com";
  set bereq.http.x-amz-content-sha256 = "UNSIGNED-PAYLOAD";
  
  # CDN signs the request using IAM role - no secret in config
  call aws_v4_sign(
    region = "us-east-1",
    service = "s3",
    access_key = var.aws_access_key,
    secret_key = var.aws_secret_key
  );
  
  # Normalize headers that break Azure/GCS compatibility
  unset bereq.http.x-amz-storage-class;
}

How S3, GCS, Azure, and Custom Stores Compare in Practice

When you're choosing your first origin storage, don't get lost in feature lists. For a CDN origin, three things really matter: time to first byte (TTFB) on a cache miss, how much you pay for requests and egress, and how the store behaves under errors. In my experience, the network distance between your CDN's shield PoP and the storage region dominates TTFB more than the provider's brand. An S3 bucket in us-east-1 fetched from a shield in Virginia will consistently beat the same bucket fetched from Frankfurt by 60-80 ms. Pick the bucket region closest to your CDN's origin shield, not just closest to your office.

Cost is where beginners get surprised. All three clouds charge you per 10k GET requests and per GB egress to the CDN. Egress is usually the bigger number, but request costs add up fast if your cache hit ratio is low and you're serving lots of tiny assets. GCS and S3 have similar list prices, Azure can be cheaper for hot-tier reads but charges more for write operations. Custom stores like MinIO running on your own hardware flip the model — you pay for disks and bandwidth upfront, but zero per-request fees, which can be a win if you're serving billions of small icons or manifests.

Finally, watch the error semantics. S3 returns fairly standard 403/404/503 codes, GCS adds 429s when you hammer it too hard, and Azure loves to return 500s with XML bodies that don't parse like S3 errors. Your CDN should handle all of them with retries and stale-while-revalidate, not just pass them to the user. These nuances aren't obvious from the marketing pages, but they're the difference between a resilient setup and one that shows users an XML error page.

Putting It Into Production: Authentication, Origin Shielding, and Cache Keys

Getting a bucket wired up as an origin is the easy part — keeping it secure and fast in production is where teams get tripped up. You’ll never want to make that S3 or GCS bucket public. Instead, lock it down and let only your CDN pull from it. On AWS that means Origin Access Control (OAC) with SigV4 signing, on GCS it’s typically a service account with HMAC keys or short-lived signed URLs, and on Azure it’s a SAS token or Managed Identity tied to the CDN’s enterprise app. We’ve seen too many origins leak because someone left “Block all public access” off for a hotfix and forgot to re-enable it. Don’t do that — verify access with a direct curl to the origin URL; you should get a consistent 403 without the CDN’s headers.

Once auth is solid, turn on Origin Shield immediately. Without it, a cache miss in New York, London, and Singapore means three separate hits to your bucket. With Shield enabled in a central region — say, us-east-1 or europe-west1 — those three misses collapse into one shield miss and two shield hits. In practice, that cuts origin egress by 80-93% and drops your origin request rate from thousands per second to a few dozen during peak. It also smooths out latency spikes when your object store throttles; S3 will start returning 503 SlowDown around 5,500 GETs per second per prefix, and Shield is the cheapest way to stay well under that limit.

The last piece is cache key hygiene. Object stores are case-sensitive and query-string sensitive, so /video/1080p.mp4?token=abc and /video/1080p.mp4?token=xyz are two different cache keys by default even though they point to the same object. Normalize what you don’t need. Strip marketing params, sort the rest, and lowercase the path if your app allows it. And tune your timeouts aggressively — we usually start with a 2s connect timeout, 8s read timeout, and 15s total timeout for storage origins. If you’re seeing tail latency from cold storage tiers, check our Origin Timeout Configuration Guide: Connect, Read, and Total Timeout Tuning before you just bump everything to 30s.

# Production-ready CloudFront + S3 OAC (Terraform) with Shield
resource "aws_cloudfront_distribution" "cdn" {
  origin {
    domain_name = aws_s3_bucket.origin.bucket_regional_domain_name
    origin_id   = "s3-primary-origin"

    origin_access_control_id = aws_cloudfront_origin_access_control.oac.id

    # Enforce SSL verification back to S3
    custom_origin_config {
      origin_ssl_protocols = ["TLSv1.2"]
      origin_protocol_policy = "https-only"
    }
    connection_timeout = 2  # seconds
    connection_attempts = 2
  }

  # Origin Shield in the bucket's region
  origin_shield {
    enabled              = true
    origin_shield_region = "us-east-1"
  }

  default_cache_behavior {
    target_origin_id = "s3-primary-origin"
    viewer_protocol_policy = "redirect-to-https"
    allowed_methods = ["GET", "HEAD"]
    cached_methods  = ["GET", "HEAD"]
    
    # Normalize cache key: ignore token, keep version
    cache_policy_id = aws_cloudfront_cache_policy.normalized.id
  }
}

# Cache policy: whitelist only 'v' query param
resource "aws_cloudfront_cache_policy" "normalized" {
  name = "normalize-storage-keys"
  parameters_in_cache_key_and_forwarded_to_origin {
    query_strings_config { query_string_behavior = "whitelist" 
      query_strings { items = ["v"] }
    }
    headers_config { header_behavior = "none" }
    cookies_config { cookie_behavior = "none" }
  }
  min_ttl     = 86400  # 1 day
  default_ttl = 604800 # 7 days
  max_ttl     = 2592000
}

Advanced Patterns: Failover, Hedged Requests, and Debugging Those Maddening 403s

When you’re serving at scale, a single object store in a single region isn’t enough. We run active-passive failover for storage origins everywhere now. Primary is S3 in us-east-1, secondary is a replicated bucket in eu-west-1 or a completely different store like GCS. Your CDN should health-check the primary every 5-10 seconds and fail over after 2 consecutive 5xx or timeout failures within 15 seconds. Don’t use 403 or 404 to trigger failover — you’ll just replicate a bad key to the secondary and double your errors. Keep failover strictly for 500, 502, 503, 504, and timeouts. For even tighter tail latency, we’re also hedging requests: send the same GET to both primary and shield-secondary after a 50ms delay and use whichever returns first. It sounds wasteful, but it cuts p95 origin fetch time from 420ms to under 85ms when the storage tier is under load. We break down the pattern in Origin Request Hedging: Sending Parallel Requests to Cut Tail Latency.

Most “random” 403s from S3/GCS aren’t random at all. If you’re seeing SignatureDoesNotMatch or RequestTimeTooSkewed for 0.2% of requests, check three things. First, clock skew — your CDN edge and your signing service must be within 5 minutes, ideally NTP-synced to within 500ms. Second, header canonicalization — if you’re signing via a Worker or Lambda@Edge, make sure you’re not altering host or x-amz-content-sha256 after signing. Third, URL encoding — a space encoded as + versus %20 will break the signature. For custom S3-compatible stores like MinIO or Ceph, we also see 403s when they don’t support chunked transfer encoding; force Content-SHA256 to be UNSIGNED-PAYLOAD or disable chunking at the CDN.

Finally, don’t skip origin TLS verification. It’s tempting to set “verify origin SSL: off” to fix a cert chain issue, but you’ve just opened yourself to a man-in-the-middle between edge and origin — especially with custom stores using Let’s Encrypt certs or private CAs. Always upload your CA bundle to the CDN, enforce TLS 1.2+, and verify the hostname matches the cert’s SAN. If your custom store sits behind a private ALB, use mTLS with a client cert pinned at the origin. We cover the full setup, including what to do when Azure Blob rotates its intermediate cert without warning, in Origin SSL Verification at CDN Edges: Preventing Man-in-the-Middle Attacks.

Frequently Asked Questions