CDN Log Aggregation Architecture: Collecting Billions of Events Without Dropping Data

Back in 2019 we pushed a config change that doubled our log volume overnight, and by 2 AM our central syslog aggregator was 14 hours behind while edge disks were filling at 800 MB/s. We weren't missing alerts because we had no traffic — we were missing them because we couldn't collect what we'd already generated. If you've ever watched a CDN drop 30% of your access logs during peak and then tried to explain to security why you can't trace an attack, you know the problem isn't generating events, it's moving billions of them without losing a single one.

Why Naive Disk Buffering Collapses at Billions of Events

Most teams start with what I did ten years ago: rsyslog or fluentd tailing Nginx access logs, writing to local disk, then forwarding with TCP to a central collector. It works beautifully until you cross about 50k requests per second per PoP. At that point, file rotation, inode contention, and fsync latency turn your "simple" pipeline into a data shredder. I've seen PoPs where the kernel was spending 40% of its time in iowait just trying to keep up with log writes, while the NIC was barely at 20% utilization. The bottleneck was never the network — it was the assumption that disks are infinite and fast.

The second failure mode is subtler. When your CDN Real-Time Log Streaming Architecture: From Edge Events to Analytics in Seconds relies on in-memory queues with no backpressure signal, a single slow downstream consumer causes silent drops. We learned this the hard way during a Black Friday sale for a large e-commerce customer. Our Kafka cluster in us-east-1 paused for a 90-second leader election, and our edge shippers, configured with a 10,000-event memory queue, just started discarding new events. We lost 18 million log lines — including the exact checkout failures the customer needed to debug. That incident taught us that a pipeline that drops data under pressure isn't a pipeline, it's a liability.

What actually works at scale is treating every edge node as an ephemeral, unreliable producer that must never trust the network. You need memory-mapped, disk-backed buffers with explicit durability guarantees and a shipper that blocks or sheds load visibly — not silently. We moved to Vector and later to a custom Rust shipper that uses a write-ahead log (WAL) on NVMe, checksummed and fsynced in batches, not per event. The HTTP Working Group Specifications remind us that even HTTP semantics assume reliable delivery — your logging pipeline should be stricter than the protocol it observes.

# vector.toml - edge shipper with durable disk buffer
[sources.nginx_access]
type = "file"
include = ["/var/log/nginx/access.json"]
read_from = "beginning"

[sinks.kafka_edge]
type = "kafka"
inputs = ["nginx_access"]
bootstrap_servers = "kafka-edge.service.consul:9092"
topic = "cdn.raw_logs"
compression = "lz4"
batch.max_events = 5000
batch.timeout_secs = 1.0

# Critical: durable buffer on NVMe, not memory
[sinks.kafka_edge.buffer]
type = "disk"
max_size = 107374182400  # 100GB - ~4 hours at 1.2M rps
when_full = "block"      # never drop, apply backpressure to source

[transforms.enrich]
type = "remap"
inputs = ["nginx_access"]
source = '''
  .pop_id = get_env_var("POP_ID")
  .edge_timestamp_ns = to_unix_timestamp(now(), unit: "nanoseconds")
  .ingest_attempt = 1
'''

Designing a Backpressure-Aware Event Collection Pipeline

Once you've hardened the edge, the core problem becomes flow control. You can't let 250 PoPs each pushing 500k events per second overwhelm your central aggregation tier. The trick isn't to scale the center infinitely — it's to make backpressure a first-class citizen all the way from Kafka to S3. We've built our Clickstream Analysis at CDN Edge: Understanding User Navigation Patterns pipeline to propagate pressure backwards: if our ClickHouse loaders lag, Kafka consumer lag grows, which throttles the edge shippers, which in turn applies TCP backpressure to Nginx's log module. It sounds scary to block, but blocking is observable and recoverable — dropping is neither.

At the transport layer, that means idempotent producers with strong acks. Don't accept `acks=1` for CDN logging. You need `acks=all` with `enable.idempotence=true` and `max.in.flight.requests=5`. Yes, it adds 8-12ms of latency per batch, but it eliminates duplicates and guarantees that a batch isn't considered delivered until a quorum of brokers has it. We also set `retries` to effectively infinite with exponential backoff capped at 30 seconds, plus a circuit breaker after 5 minutes that spills to local disk WAL instead of dropping. One lesson from running this for three years: retries without jitter will thundering-herd your brokers back into failure. We use full jitter and it cut our retry storms by 70%.

For the last mile — from Kafka to object storage or analytics — we use exactly-once semantics via transactional sinks. A common pattern we run in production splits streams: hot path to ClickHouse for real-time alerting, cold path to S3/Parquet for batch analytics. We checkpoint offsets only after S3 multipart uploads complete and are validated with ETags. If validation fails, we don't commit. That guarantees we never advance past data that isn't durably stored. The web.dev Articles on reliable beaconing helped shape our thinking on retry guarantees for client-side events, and the same principles apply server-side at 1000x scale.

// Go shipper - reliable Kafka producer with backpressure and WAL fallback
producer, _ := kafka.NewProducer(&kafka.ConfigMap{
    "bootstrap.servers":  "kafka.service.consul:9092",
    "acks":               "all",
    "enable.idempotence": true,
    "compression.type":   "lz4",
    "retries":            2147483647,
    "retry.backoff.ms":   100,
    "queue.buffering.max.messages": 500000,
    "queue.buffering.max.ms":       1000,
    "message.timeout.ms":           300000,
})

deliveryChan := make(chan kafka.Event, 10000)
err = producer.Produce(&kafka.Message{
    TopicPartition: kafka.TopicPartition{Topic: &topic, Partition: kafka.PartitionAny},
    Value:          payload,
    Headers:        []kafka.Header{{Key: "pop_id", Value: []byte(popID)}},
}, deliveryChan)

// Block on delivery report - apply backpressure if Kafka is slow
e := <-deliveryChan
m := e.(*kafka.Message)
if m.TopicPartition.Error != nil {
    wal.Append(payload) // spill to disk WAL, don't drop
    metrics.Inc("shipper_wal_spill_total")
} else {
    metrics.Observe("shipper_ack_latency_ms", time.Since(start).Milliseconds())
}

Measuring Losslessness: Throughput, Latency, and Durability Benchmarks

You can't claim "we don't drop data" without proving it under failure. We run continuous chaos tests where we kill 30% of Kafka brokers, throttle S3 to 50 MB/s, and fill edge disks to 85% during synthetic peak loads. The goal is to measure two things: how long you can sustain ingest without loss, and how fast you recover without creating duplicates. Our current generation sustains 2.8 billion events per hour globally with less than 15 seconds of end-to-end p99 latency from edge event to queryable in ClickHouse, even when a full region is degraded.

Cost is the other hidden metric. Full-fidelity logging at that scale is expensive, and that's where sampling has to be intentional, not accidental. We don't randomly drop 10% at the edge when overloaded — we make explicit sampling decisions upstream. For our detailed analysis, see CDN Log Sampling Strategies: Reducing Storage Costs Without Losing Signal — we've found that tiered sampling (100% for 4xx/5xx and WAF events, 10% for 2xx static assets) cuts storage by 62% while preserving the signals we actually alert on. The key is that sampling must be a policy, not a side effect of an overloaded queue.

When we evaluate any new data pipeline component, we score it against a strict set of production service level objectives. If it can't meet these for 7 days straight under peak load, it doesn't ship. These aren't synthetic benchmarks — they're taken from our primary pipeline serving 180 Tbps of egress last quarter.

Putting It Together: A Tiered Buffer That Actually Holds at 5M RPS

The theory's nice, but here's what we run in production across 180+ PoPs handling 4.8-5.2M requests per second at peak. We use a two-tier buffer: a bounded in-memory ring (typically 256MB or ~2.1M log lines) fronting a memory-mapped disk spill. The in-memory tier gives you sub-millisecond ingest — we see p99 write latency of 0.3ms — while the mmap'd file gives you durability without paying the fsync tax on every event. The trick is we don't fsync synchronously. We batch fsyncs every 250ms and only block producers when both tiers hit 85% capacity. That threshold is critical; if you wait until 100%, you've already started dropping.

Your collector should never use an unbounded channel or a naive log file append. Instead, wire explicit backpressure signals from your shipper (Vector, Fluent Bit, or a custom Go collector) back to the edge worker. When our Kafka tier in the region goes slow — say, brokers hit 70% I/O wait during a compaction — the shipper applies `when_full = block` rather than `drop_newest`. Nginx/OpenResty will then pause reading from the shared memory log buffer for 10-50ms instead of overwriting. You'll add 5-15ms to TTFB for that 0.2% of requests, but you won't lose the 40,000 events per second that'd otherwise vanish. That trade-off is what makes User Journey Analytics from CDN Logs: Mapping the Path to Conversion reliable — you can't reconstruct a journey if you're missing 2% of pageviews in the middle of the funnel.

# vector.yaml - backpressure-aware CDN log pipeline
sources:
  cdn_edge_logs:
    type: file
    include: ["/var/log/nginx/access.json"]
    read_from: beginning

transforms:
  enrich:
    type: remap
    inputs: ["cdn_edge_logs"]
    source: |
      .pop = get_env_var("POP_ID")
      .ingest_ns = now()

sinks:
  kafka_primary:
    type: kafka
    inputs: ["enrich"]
    bootstrap_servers: "kafka.region.internal:9092"
    topic: "cdn.logs.raw"
    compression: lz4
    batch.max_bytes: 1048576
    buffer:
      type: disk
      max_size: 107374182400  # 100GB spill per node
      when_full: block        # NEVER drop_newest at edge
    request.timeout_secs: 30
    acknowledgements.enabled: true

Advanced Patterns: Hot Shards, Clock Skew, and the Silent Drops You Don't See

Once you've solved basic durability, the nasty failures start. The first is hot shards. At billions of events, a simple `hash(client_ip)` partition strategy will hammer one Kafka partition if a single /16 starts scraping you. We've seen a single bad botnet push 340k rps to one partition while the other 23 sat at 12k. The fix is two-level partitioning: hash by `ray_id` or `request_id` for even distribution, then add a secondary routing key for analytics. And always pre-create partitions with headroom — we run 48 partitions per topic and never let a single partition exceed 180k messages/sec or 85MB/s. If you do, your p99 end-to-end latency jumps from 1.2s to 8-9s and your consumers start lagging by millions.

Second killer: clock skew. Your 180 PoPs don't have perfect NTP. We've measured 45-120ms drift between edge nodes, which destroys event ordering if you're relying on edge timestamps for sessionization. Don't trust `time_local` from Nginx. Inject a monotonic `ingest_ns` at the collector and use the Kafka broker timestamp as the source of truth, then re-order with a 2-second watermark window in Flink/ClickHouse. Without that, your Conversion Attribution from CDN Edge: First-Party Tracking Without Client-Side Scripts will misattribute 3-5% of conversions simply because the checkout event arrived with an earlier wall-clock time than the product view.

Finally, watch for silent drops in your "lossless" pipeline. The usual culprit isn't Kafka — it's file rotation. If logrotate does a `copytruncate` while Vector is tailing, you'll lose 200-800ms of logs with zero errors in your metrics. Switch to `create` with `postrotate` sending `USR1` to Nginx, and monitor `vector_buffer_discarded_events_total` and `kafka_producer_record_errors` as SLI burn-rate alerts, not just throughput. We also run a continuous canary: each PoP emits a synthetic log every second with a UUID, and a validator checks it arrived in ClickHouse within 10s. If canary loss exceeds 0.001% over 5 minutes, we page. This is how we caught a kernel bug where `mmap` pages weren't flushing under 95% disk pressure — and it's essential if you're doing Bot Traffic Analysis and Classification from CDN Logs: Separating Good from Bad, where missing even 0.1% of bot bursts makes your classifier blind to low-and-slow scrapers.

Frequently Asked Questions