CDN Log Storage Cost Management: Sampling, Aggregation, and Retention Strategies

I still remember the Slack message from finance in late 2021: "Why did our log storage bill double last month?" We'd pushed a new video product, edge traffic jumped 40%, and nobody had thought about what 300 million extra daily log lines would do to our S3 and BigQuery costs. That $18k surprise turned into a $42k quarterly overrun, and it wasn't even the CDN bandwidth that hurt us — it was storing every single request forever. If you're not actively managing your CDN logs, you're probably paying to store data you'll never query.

Why Full-Fidelity Logs Become a Cost Problem So Fast

Most teams enable CDN logging and forget about it. You get a firehose of JSON or W3C-formatted lines — timestamp, PoP, cache status, bytes, user-agent, edge latency — for every single request. At small scale that feels harmless. At 50,000 requests per second, you're generating around 4.3 billion log events per day. Even with compressed JSON at ~400 bytes per line, that's over 1.6 TB of raw logs daily, before replication or indexing.

I've worked with teams who treated log retention like backups: keep everything for 365 days just in case. That approach doesn't scale when your CDN Egress Cost Modeling: Understanding and Reducing Your Largest Bill Component is already under pressure. Log storage cost isn't just S3 or GCS pennies per GB. It's ingestion into your warehouse, indexing for search, cross-region replication, and the compute you pay to scan it all with every troubleshooting query. I audited one e-commerce client paying $11k/month for logs where 94% of queries only touched the last 72 hours.

The math gets ugly because CDN log cost scales linearly with traffic but your insight doesn't. You don't need a 200 OK for a cached 2KB image from three months ago with the same urgency as a 5xx spike from last night. Understanding which logs actually drive decisions is the first step in retention management. For standard HTTP semantics, it's helpful to ground your field selection in what HTTP Working Group Specifications actually defines as essential versus optional.

# Quick cost estimation for CDN log storage
# Assumptions: 500M req/day, 450 bytes avg log line (gzipped)

requests_per_day = 500_000_000
bytes_per_log = 450
daily_volume_gb = (requests_per_day * bytes_per_log) / (1024**3)
monthly_volume_tb = (daily_volume_gb * 30) / 1024

# Example pricing (S3 Standard + Athena scans)
storage_cost_per_gb = 0.023
ingest_and_scan_cost_per_tb = 5.00

monthly_storage = monthly_volume_tb * 1024 * storage_cost_per_gb
monthly_query = monthly_volume_tb * ingest_and_scan_cost_per_tb * 4  # avg scans

print(f"Daily: {daily_volume_gb:.1f} GB/day")
print(f"Monthly raw: {monthly_volume_tb:.2f} TB")
print(f"Est. storage: ${monthly_storage:,.2f}")
print(f"Est. query: ${monthly_query:,.2f}")
# Output: ~209 GB/day, ~6.1 TB/month, ~$143 storage + ~$122 query
# Add replication/indexing and you're at $800+/month before optimization

Building a Sampling Strategy That Preserves Signal

A good sampling strategy isn't about randomly dropping 90% of your logs and hoping for the best. That's how you miss the one error pattern that matters. What we've done on high-traffic properties is tiered, conditional sampling — keep 100% of what hurts, sample what doesn't. Errors, cache MISSes that hit origin, requests over a latency threshold, and anything with a 4xx/5xx status get kept unconditionally. Successful, cached 2xx responses for static assets? Those are perfect candidates for head-based sampling at 1:10 or even 1:100.

Tail-based sampling is more powerful if your pipeline supports it, but it's also more complex. Instead of deciding at the edge, you buffer a window of requests per trace or session and make a decision after you see the outcome. If a user session contains an error, you keep the whole session. If it's just ten clean cache HITs for images, you keep one representative log. This gives you much better debuggability than naive random sampling, especially for intermittent issues. I helped a media customer cut their log volume by 82% with conditional sampling while still catching 99.8% of error sessions — their mean time to detect origin failures didn't change at all.

Don't sample before you enrich. Tag logs at the edge with the fields you'll filter on later: cache status, PoP ID, edge compute result, and a hashed session ID. That way your downstream sampling logic can make smart decisions without re-parsing. This is also where CDN Request Pricing Analysis: Reducing Per-Request Costs at Scale ties in directly, because your sampling rate should reflect request value. Web performance guidance from web.dev Articles is useful here too — you might keep 100% of logs for LCP-related document requests but heavily sample font and image beacons.

// Cloudflare Workers / Fastly Compute@Edge style sampling logic (pseudo-JS)
function shouldSampleLog(request, response) {
  // Always keep errors and origin fetches
  if (response.status >= 400) return { keep: true, reason: "error" };
  if (response.headers.get("x-cache") !== "HIT") return { keep: true, reason: "miss_or_origin" };
  if (parseInt(response.headers.get("server-timing") || "0") > 800) return { keep: true, reason: "slow_edge" };

  // Sample successful cache hits for static assets
  const isStatic = /\.(jpg|png|css|js|woff2)$/.test(request.url);
  if (isStatic && response.status === 200) {
    return { keep: Math.random() < 0.05, reason: "sampled_static_5pct" }; // 1:20
  }

  // Default: 10% sampling for remaining 2xx HITs
  return { keep: Math.random() < 0.10, reason: "sampled_default_10pct" };
}

Aggregation and Tiered Retention: Store Less, Query Faster

Sampling reduces volume, but aggregation is what turns logs from a storage liability into a usable dataset. You don't need 500 million raw JSON lines to answer "what was my cache hit ratio per PoP in the last hour?" You need one aggregated row per PoP per minute with counts, percentiles, and error rates. We typically run a streaming job — Kinesis Firehose, Pub/Sub to Dataflow, or Kafka to ClickHouse — that rolls raw logs into 1-minute and 5-minute aggregates within seconds of ingestion. Raw logs can then be moved to cold storage or deleted much sooner.

My standard retention management setup these days is three tiers. Hot: raw or lightly sampled logs in a queryable store like ClickHouse or BigQuery for 3-7 days. Warm: 1-minute aggregates with p50/p95 latency, status code breakdowns, and byte counts retained for 30-45 days. Cold: 5-minute or hourly aggregates plus daily rollups kept for 13 months for capacity planning and year-over-year comparisons. That last tier is tiny — often less than 2% of the raw volume — but it answers 80% of the business questions finance actually asks. It also keeps you from paying for a year of raw logs when your CDN Bandwidth Commitment Pricing: When Committed Plans Save Money vs Pay-As-You-Go reviews only need hourly trends.

The key is defining aggregations before you need them. If you wait until you're trying to cut costs, you'll be scared to delete anything. Work backwards from your dashboards and alerts: cache hit ratio, origin offload, error rate by PoP, p95 TTFB, top 100 URLs by origin bytes. If a query can be answered from aggregates, it shouldn't require scanning raw logs. We keep raw logs for 7 days not because we query them constantly, but because that's our window for deep forensic debugging when aggregates show an anomaly.

Putting Sampling and Tiered Retention Into Production

Architecture diagrams are nice, but the real savings come from wiring sampling and aggregation directly into your log pipeline so you don't have to think about it at 2 a.m. during a traffic spike. We typically push this logic as close to the edge as possible — don't ship everything to your central bucket and then try to filter it. If you're running CloudFront, Fastly, Akamai, or Cloudflare, use their native log controls to decide what leaves the edge in the first place. For everything else, a lightweight shipper like Vector or Fluent Bit in front of Kinesis, Pub/Sub, or Kafka gives you deterministic control.

The pattern that works reliably at scale is: sample at the edge, aggregate in stream, then route to tiered storage. Keep 100% of your 4xx/5xx and cache MISS logs — that's your error signal. For 2xx cache HITs, which are often 85-92% of total volume, a 1:10 or even 1:100 sample is plenty for performance analysis. Then roll the sampled stream into 1-minute aggregates (p50/p95/p99 latency, request count, bandwidth, cache hit ratio) by PoP, URL pattern, and status class. That aggregation alone gives you a 30:1 to 40:1 reduction before anything hits object storage. We’ve seen a customer doing 8 TB/day of raw CDN logs drop to ~210 GB/day of sampled + aggregated data without losing any alertable signal.

# vector.toml - sampling at edge, aggregation in stream
[sources.cdn_edge_logs]
type = "http_server"
address = "0.0.0.0:8080"

[transforms.route_and_sample]
type = "route"
route.errors = '.status >= 400 || .cache_status == "MISS"'
route.success_sampled = '(.status == 200 && .cache_status == "HIT" && sample(10)) == true'
# sample() keeps 1 in 10 -> 10% sampling for healthy HITs
# errors and MISSes are always kept (100%)

[transforms.aggregate_1m]
type = "aggregate"
inputs = ["route_and_sample.success_sampled", "route_and_sample.errors"]
interval_ms = 60000
function.count = "count"
function.p95_latency = "quantile(0.95)"
group_by = ["pop", "status_class", "cache_status", "url_pattern"]

[sinks.s3_hot]
type = "aws_s3"
inputs = ["route_and_sample.errors"] # raw errors -> hot storage
bucket = "cdn-logs-hot"
compression = "zstd"
encoding.codec = "json"

[sinks.s3_warm_aggregates]
type = "aws_s3"
inputs = ["aggregate_1m"]
bucket = "cdn-logs-warm-aggregates"
compression = "zstd"
# Lifecycle: hot (7d) -> warm/S3 Standard-IA (30d) -> cold/Glacier (365d)

Once routing is set, automate your lifecycle. Don't rely on manual bucket cleanup. Set a 7-day retention for raw sampled logs in Standard, transition aggregates to Standard-IA after 30 days, and push anything older than 90 days to Glacier Instant or Deep Archive for compliance. Pair this with columnar compression — converting JSON to Parquet + Zstandard will cut storage another 60-75% and make Athena/BigQuery scans 10x cheaper. If you haven't tuned your storage classes yet, review CDN Storage Cost Optimization: Tiering, Compression, and Lifecycle Policies — getting the tiering wrong can cost more than the logs themselves. For pipelines that aggregate at the edge, doing that work in a Worker or Edge Function can be cheaper than central stream processing, but watch your invocation costs as covered in CDN Edge Compute Cost Optimization: Function Pricing Models and Efficiency Tricks.

Advanced Patterns: Adaptive Sampling, Rehydration, and Fixing Blind Spots

Static 1:10 sampling works until it doesn't. The biggest gotcha we see is tail-latency and low-frequency errors getting washed out. A 0.01% error rate on 50 billion requests/day is still 5 million errors — but if you sampled those 200s away uniformly, your p99 will look artificially smooth. The fix is adaptive sampling. Instead of a fixed rate, adjust sampling based on signals: keep 100% when latency > 1.5x your SLO, when a PoP's error rate spikes >0.5%, or when you see a new URL pattern or bot signature. Most stream processors let you do this with a simple lookup table or feature flag. It'll bump your ingest by 15-20% during incidents — exactly when you want more data — and stay lean the rest of the time.

The second pattern is rehydration on demand. You'll occasionally need full-fidelity for forensics — say, a cache poisoning attempt or a billing dispute. You don't need to store it hot for that. Keep a 24-48 hour buffer of unsampled logs on cheap local NVMe or R2/S3 Standard, then let it age out, but also emit a parallel ultra-sampled stream (1:1000) to cold storage for a year. If you need to investigate something older, replay from the CDN provider's short-term raw archive — Fastly and Cloudflare both offer 3-7 day raw log re-push via API — or temporarily raise sampling to 100% for the affected pattern. We've used this to reconstruct a full attack chain from just 1:1000 cold samples plus 1 hour of rehydrated raw logs.

When sampling breaks your debugging, it's usually one of three things. First, cardinality explosion in your aggregates — grouping by full URL instead of pattern creates millions of series and kills your query engine. Normalize to route patterns (/product/:id, /api/v2/*) before aggregating. Second, clock skew across PoPs makes your 1-minute buckets misalign; standardize on NTP-synced edge timestamps, not collector arrival time. Third, inconsistent sampling decisions across layers — if the CDN samples and your shipper samples again, you get non-deterministic gaps. Make sampling authoritative in one place. And if your log volume is predictable enough to forecast quarterly, look at locking in ingest and storage pricing — CDN Reserved Capacity Pricing: Locking In Rates for Predictable Workloads can offset the baseline cost of that 7-day hot buffer.

Frequently Asked Questions