CDN Real-Time Log Streaming Architecture: From Edge Events to Analytics in Seconds

At 2:13 AM on Black Friday, our status page was green while customers were seeing broken images across the entire US East Coast. Origin was throwing 502s on about 18% of requests, but our batch log delivery to S3 was still on its 10-minute cycle. By the time the logs landed and we spotted the bad cache-key pattern, we'd served over half a million errors. That's the night I stopped believing batch uploads were good enough for anything operational.

Why Batch Delivery Can't Keep Up With Edge Events

For years, the standard approach to CDN logging was simple: write access logs to local disk on each edge node, rotate them every few minutes, then push them to object storage. It works fine if you're doing next-day billing reports or long-term trend analysis. But if you're trying to do actual Clickstream Analysis at CDN Edge: Understanding User Navigation Patterns or detect an attack in progress, that 5 to 15 minute delay is an eternity. I've seen teams try to shorten the batch window to 60 seconds to get closer to real-time logging, and all they got was a storm of tiny files, S3 PUT throttling, and gaps when a PoP got busy.

The core issue is how edge events behave under load. When traffic spikes, that's exactly when you need visibility most, and it's also when disks fill, compressors stall, and upload agents fall behind. In 2018 we had a PoP in Frankfurt that handled around 1.2M requests per second during a software launch. The batch uploader couldn't keep up and started dropping rotated files to avoid filling the disk. We didn't lose billing data — but we lost the precise error stream we needed to debug a 1% cache poisoning issue. Real-time log streaming solves this by never letting logs sit still in the first place. Instead of storing and forwarding, you capture the request as a structured event and push it into a durable stream within milliseconds.

That shift changes how you think about the format, too. You aren't writing Apache Common Log Format to a file anymore. You're emitting a self-contained JSON event per request at the moment the response leaves the edge. Here's the kind of minimal structure I push from Varnish/Fastly-style edges today — it includes just enough to power CDN analytics without bloating the stream:

{
  "timestamp": "2024-11-15T02:13:42.184Z",
  "edge_pop": "iad-042",
  "request_id": "7f3a9c1e-4b5d-4a8e-9c1d-2e5f6a7b8c9d",
  "client_ip": "203.0.113.45",
  "method": "GET",
  "url": "/api/product/98412/images/hero.webp",
  "status": 502,
  "cache_status": "MISS",
  "response_time_ms": 47,
  "bytes_sent": 0,
  "user_agent": "Mozilla/5.0 ...",
  "trace_parent": "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01"
}

How Edge Events Travel From Memory to Analytics in Seconds

A proper log streaming pipeline has four stages, and the goal is to keep data moving without ever touching disk unless you have to. First, the edge worker emits the event directly from memory — no file involved. We use shared-memory buffers or an embedded shipper like Vector or Fluent Bit that tails the event bus. Second, that shipper forwards to a regional aggregator over persistent HTTP/2 or gRPC connections, which handles backpressure. If you try to open a new TLS connection per log line, you'll melt your edge CPU. The HTTP Working Group Specifications on persistent connections and header compression are worth reading here because they explain why connection reuse is non-negotiable at this scale.

Third is the central streaming backbone. Most teams I work with now use Kafka, Redpanda, or Kinesis for this — it doesn't matter as much as you think, as long as it can handle partitioned, ordered streams and replay. Each PoP maps to a partition key so you keep per-PoP order but still scale horizontally. From there, events fan out to multiple consumers: one for real-time alerting, one for your data lake, and one for downstream CDN analytics. We've kept end-to-end latency — from the moment status: 502 is set to the moment it appears in ClickHouse — under 3 seconds at 2M events per second. You don't need that for every use case, but you want the headroom. For a deeper look at scaling the collection layer, see CDN Log Aggregation Architecture: Collecting Billions of Events Without Dropping Data.

The last piece people forget is the client. If your log consumer is a serverless function that cold-starts on every batch, you've just reintroduced the lag you were trying to remove. We run a small fleet of Vector aggregators that do lightweight transforms and ship to ClickHouse and S3 in parallel. They sample, redact PII, and enrich with GeoIP before the data ever hits storage. That preprocessing is where you save real money. CDN Log Sampling Strategies: Reducing Storage Costs Without Losing Signal covers how we sample 1-in-100 of 2xx image hits but keep 100% of 5xx and cache MISS events. Here's a simplified Vector topology we use at the regional layer to keep log streaming reliable without writing to disk:

[sources.edge_events]
type = "datadog_agent" # receives JSON over HTTP from edge shippers
address = "0.0.0.0:8282"

[transforms.enrich_and_filter]
type = "remap"
inputs = ["edge_events"]
source = '''
  .pop_latency_ms = to_int!(.response_time_ms)
  if .status >= 500 {
    .severity = "critical"
  } else if .cache_status == "MISS" {
    .severity = "info"
  }
  # drop health checks, keep all errors
  if .url == "/healthz" && .status == 200 { abort }
'''

[sinks.kafka_stream]
type = "kafka"
inputs = ["enrich_and_filter"]
bootstrap_servers = "kafka-regional.internal:9092"
topic = "cdn.edge_events"
compression = "lz4"
batch.max_bytes = 524288
batch.timeout_secs = 0.5

[sinks.clickhouse_realtime]
type = "clickhouse"
inputs = ["enrich_and_filter"]
endpoint = "https://clickhouse.internal:8123"
database = "cdn_logs"
table = "edge_events_realtime"
batch.max_events = 10000
batch.timeout_secs = 1

What Seconds-Late Actually Means: Latency, Loss, and Cost

When vendors say "real-time," I always ask: real-time for who? There's a huge difference between a stream that delivers 95% of events in 2 seconds and one that delivers 99.9% in 2 seconds. That last 0.9% is where your tail latency lives, and it's usually caused by retry queues and consumer lag. We learned this the hard way when our alerting on origin latency was firing 30 seconds late during a DDoS — not because the edge was slow, but because our consumer group was rebalancing. Now we monitor consumer lag as a first-class metric and alert if any partition lags by more than 2 seconds, not just if the pipeline is down.

You also have to be honest about at-least-once vs. exactly-once. For CDN analytics, at-least-once with idempotent request IDs is almost always the right call. Trying to guarantee exactly-once across global PoPs will double your latency and cost. We deduplicate on request_id in ClickHouse using a ReplacingMergeTree, which gives us effectively once without slowing the hot path. Guidance from the web.dev Articles on using Server-Timing and traceparent headers has also helped us correlate edge events with browser performance data, so we can join real user metrics with log streaming data without an extra lookup.

If you're pitching this to leadership, don't talk about architecture — talk about the trade-offs in numbers they can compare. I've run both batch and streaming side-by-side for months, and the differences are stark. You don't need zero latency everywhere, but you should know what you're paying for each second you shave off.

Putting It Together: A Production Streaming Stack That Actually Scales

We've seen too many teams try to bolt real-time streaming onto a batch-oriented CDN setup and wonder why it falls over at 50k requests per second. You can't just tail log files and hope Kafka keeps up. In production, you need an agent on every edge node that never touches disk unless it has to. We typically run Vector or Fluent Bit as a sidecar with a memory buffer of 10-20MB — just enough to survive a 2-3 second downstream stall without dropping events. It batches events into 64KB compressed payloads and pushes via persistent HTTP/2 or gRPC to a regional aggregator, not directly to your central Kafka. That extra hop cuts cross-region bandwidth by 70% and keeps p99 ingestion latency under 400ms even when a PoP is handling 80k RPS.

From there, the aggregator fans out to a central stream. Don't overcomplicate this — a managed Kafka (Confluent, MSK) or Redpanda topic with 24-48 partitions is plenty for most workloads doing 100-300k events/sec. Key your partitioning by PoP ID or ray ID, not by URL, so you don't create hot partitions during a cache purge. Your consumers should be doing lightweight enrichment — geo-IP, bot scoring, user ID stitching — before sinking to ClickHouse, Apache Druid, or BigQuery. If you're sinking to ClickHouse, use the native Kafka engine or Vector's ClickHouse sink with async inserts; we've seen it sustain 1.2M rows/sec on a 3-node cluster while keeping query latency for last-5-minute dashboards under 900ms.

# vector.toml - edge agent config for sub-second delivery
[sources.cdn_logs]
type = "file"
include = ["/var/log/cdn/access.json"]
read_from = "end"

[transforms.enrich]
type = "remap"
inputs = ["cdn_logs"]
source = '''
  .timestamp = now()
  .pop = get_env_var("POP_ID")
  # drop health checks, keep only cacheable + dynamic
  if .status == 200 && .path == "/health" { abort }
  .bot_score = to_int(.http_user_agent) # replaced by WASM classifier upstream
'''

[sinks.kafka_aggregator]
type = "kafka"
inputs = ["enrich"]
bootstrap_servers = "aggregator.us-east.internal:9092"
topic = "cdn.edge.events"
compression = "lz4"
batch.max_bytes = 65536
batch.timeout_secs = 0.25
buffer.type = "memory"
buffer.max_events = 50000
buffer.when_full = "block" # backpressure, don't drop

When Seconds Become Minutes: Backpressure, Duplicates, and Traffic Spikes

Here's where most streaming architectures silently fail: backpressure. When your analytics sink lags — say ClickHouse is merging parts during a 10x flash sale spike — you can't let the edge agent drop events. But you also can't let it OOM the edge server. The trick is tiered buffering. We configure Vector to block for up to 5 seconds, then spill to a bounded 500MB disk buffer on the edge SSD. If that fills up in under 60 seconds, you're not under-provisioned, you're under-partitioned downstream. At that point, you need autoscaling on your Kafka consumers, not a bigger edge buffer. Set alerts on `buffer_byte_size` > 80% and consumer lag > 15k messages — if lag stays above 30 seconds for more than two minutes, you've already lost your "real-time" SLA.

You'll also hit duplicates and out-of-order delivery. It's not a bug, it's TCP retries. During a regional network blip, we've seen 0.8-1.2% duplicate events for about 90 seconds after recovery. Don't try to dedupe at the edge — you'll burn CPU you need for serving. Instead, use an idempotent sink key like `request_id` + `edge_timestamp` and let ClickHouse's `ReplacingMergeTree` or Druid's exactly-once ingestion handle it. For ordering, don't rely on Kafka offset order for analytics. Always use the edge-generated `event_time`, not ingest time. If you're building User Journey Analytics from CDN Logs: Mapping the Path to Conversion, a 2-second reordering window in your Flink or Materialize view will fix 99.9% of pathing errors without adding noticeable delay.

Finally, filter aggressively before you pay to transport it. You don't need to stream every 304 Not Modified or every static asset hit to your analytics warehouse. Sample 1-in-10 for images/CSS, keep 100% for HTML/API/cart events, and do bot filtering at the aggregator layer. That one rule alone cut a customer's egress from 4.2 TB/day to 680 GB/day. That filtered stream is what you want feeding your Conversion Attribution from CDN Edge: First-Party Tracking Without Client-Side Scripts and your Bot Traffic Analysis and Classification from CDN Logs: Separating Good from Bad — clean, high-signal events, not noise. If you've got to debug, add a separate `cdn.raw_sample` topic with 0.1% sampling for full-fidelity replay.

Frequently Asked Questions