Real-Time CDN Analytics Pipeline: From Access Logs to Actionable Dashboards

CDN access logs are the richest data source for understanding how your content delivery infrastructure performs. Every request generates a log line containing the timestamp, client location, requested URL, cache status, response code, response time, bytes served, and dozens of other fields. At CDN scale — billions of requests per day — this raw data is both an invaluable operational resource and an overwhelming engineering challenge. A real-time analytics pipeline transforms this flood of log data into structured, queryable metrics that power operational dashboards, feed anomaly detection systems, and validate SLO compliance.

Pipeline Architecture Overview

A production CDN analytics pipeline has four stages: ingestion (collecting logs from edge nodes), processing (parsing, enriching, and aggregating), storage (persisting structured data for querying), and visualization (dashboards and alerting). Each stage must handle the volume and velocity of CDN-scale data while maintaining the sub-minute latency that makes the analytics “real-time” in operational terms.

Log Ingestion: From Edge to Stream

CDN edge nodes generate access logs locally and ship them to a centralized ingestion point. The shipping mechanism depends on the CDN platform: self-managed CDNs typically use a log forwarder (Fluent Bit or Vector) running as a sidecar process on each edge server, while commercial CDNs offer log push services (Cloudflare Logpush, Fastly Real-Time Log Streaming, AWS CloudFront real-time logs).

Apache Kafka serves as the ingestion buffer between edge log shippers and the processing layer. Kafka provides durable, ordered message storage that decouples producers (edge nodes) from consumers (the processing pipeline). This decoupling is critical for CDN analytics because edge nodes cannot wait for the analytics pipeline — any back-pressure on log shipping would degrade content delivery performance.

Kafka Topic Design for CDN Logs

Topic partitioning determines both parallelism and ordering guarantees. For CDN logs, partition by PoP identifier to preserve per-PoP ordering while enabling parallel processing across PoPs. With 300 edge PoPs, 300 partitions provide natural parallelism for a 300-node edge network.

# Kafka topic configuration for CDN access logs
kafka-topics --create \
  --topic cdn-access-logs \
  --partitions 300 \
  --replication-factor 3 \
  --config retention.ms=604800000 \
  --config segment.bytes=1073741824 \
  --config compression.type=zstd \
  --config max.message.bytes=1048576

# Key: PoP identifier (ensures per-PoP ordering)
# Value: JSON-encoded log line
# Compression: zstd for best ratio on structured text
# Retention: 7 days (enables pipeline replay from any
#            point in the past week)

Seven-day retention enables pipeline replay from any point in the past week. When the processing pipeline deploys a bug fix or enrichment update, operators can reprocess the entire week without needing a separate log archive. Zstd compression typically achieves 8-12x compression on structured CDN log data, reducing Kafka storage and network costs significantly.

Stream Processing: Parse, Enrich, Aggregate

The stream processor transforms raw log lines into structured, enriched records and pre-aggregated metrics. Apache Flink is the standard choice for CDN-scale stream processing because of its exactly-once semantics, event-time processing with watermarks, and efficient windowed aggregation.

Log Parsing and Enrichment

Raw CDN log lines contain IP addresses, timestamps, and identifiers that need enrichment to become analytically useful. The enrichment pipeline adds geographic context (GeoIP lookup: country, region, city, latitude/longitude), network context (ASN lookup: ISP name, network type), device context (User-Agent parsing: browser, OS, device type), and content context (URL classification: static/dynamic, content type, application segment).

GeoIP and ASN lookups use in-memory databases (MaxMind GeoIP2, IP2Location) loaded into each Flink task manager. These databases are small (100-200 MB) and provide sub-microsecond lookup times, adding negligible latency to the processing pipeline. User-Agent parsing uses a compiled regex-based classifier that identifies the top 50 browser/OS combinations, covering 99% of CDN traffic.

Windowed Aggregation

Pre-aggregating metrics during stream processing reduces the storage and query load on the analytics database by orders of magnitude. Instead of storing one row per request (billions per day), the pipeline stores one aggregated row per unique combination of dimensions per time window.

-- Conceptual aggregation output (1-minute windows)
-- Each row summarizes all requests matching the dimension combination

{
  "window_start": "2026-10-01T14:35:00Z",
  "window_end": "2026-10-01T14:36:00Z",
  "pop": "SJC",
  "country": "US",
  "cache_status": "HIT",
  "status_class": "2xx",
  "content_type": "image/webp",
  "metrics": {
    "request_count": 47823,
    "bytes_total": 2894712340,
    "latency_p50_ms": 12,
    "latency_p95_ms": 45,
    "latency_p99_ms": 128,
    "error_count": 3,
    "unique_ips_hll": "AAECAwQ..."  // HyperLogLog sketch
  }
}

The aggregation uses HyperLogLog (HLL) sketches for approximate unique counts (unique IPs, unique URLs). HLL provides cardinality estimates with 1-2% error while requiring only 12 KB of memory per counter, compared to exact counting that would require storing every unique value. For CDN analytics, where the question is “approximately how many unique users visited this PoP?” rather than the exact count, HLL’s accuracy is more than sufficient.

Storage: ClickHouse for Analytical Queries

ClickHouse is the dominant choice for CDN analytics storage because it excels at exactly the workload CDN dashboards demand: scanning billions of rows, filtering by dimensions, and computing aggregations — all in sub-second query times. Its columnar storage format reads only the columns referenced in a query, and vectorized execution processes data in CPU-cache-friendly batches.

Table Schema Design

The schema follows a star schema pattern with a central fact table (aggregated metrics) and dimension tables (PoP metadata, content classification). The fact table uses ClickHouse’s MergeTree engine family with partitioning by day and ordering by timestamp + PoP + cache status, optimizing the most common dashboard queries.

CREATE TABLE cdn_metrics (
    window_start DateTime,
    pop String,
    country LowCardinality(String),
    cache_status LowCardinality(String),
    status_class LowCardinality(String),
    content_type LowCardinality(String),
    request_count UInt64,
    bytes_total UInt64,
    latency_p50 UInt32,
    latency_p95 UInt32,
    latency_p99 UInt32,
    error_count UInt32,
    unique_ips AggregateFunction(uniq, String)
)
ENGINE = AggregatingMergeTree()
PARTITION BY toYYYYMMDD(window_start)
ORDER BY (window_start, pop, cache_status, status_class)
TTL window_start + INTERVAL 90 DAY;

-- Materialized view: hourly rollup for long-range queries
CREATE MATERIALIZED VIEW cdn_metrics_hourly
ENGINE = AggregatingMergeTree()
PARTITION BY toYYYYMM(window_hour)
ORDER BY (window_hour, pop)
TTL window_hour + INTERVAL 365 DAY
AS SELECT
    toStartOfHour(window_start) AS window_hour,
    pop,
    sum(request_count) AS request_count,
    sum(bytes_total) AS bytes_total,
    avg(latency_p50) AS latency_p50,
    max(latency_p99) AS latency_p99_max,
    sum(error_count) AS error_count,
    uniqMerge(unique_ips) AS unique_ips
FROM cdn_metrics
GROUP BY window_hour, pop;

LowCardinality(String) columns use dictionary encoding for fields with few distinct values (cache status has 5-6 values, status class has 5 values), reducing storage and query time by 2-5x compared to regular strings. The TTL clause automatically drops data older than 90 days for minute-level data and 365 days for hourly rollups, keeping storage bounded without manual cleanup.

Dashboard Design for CDN Operations

The Grafana dashboard layer transforms ClickHouse data into operational visibility. Effective CDN dashboards follow a hierarchical structure: a global overview panel at the top, drill-down by PoP or region in the middle, and detailed metric explorers at the bottom. The layout supports the operational workflow of noticing a problem (overview), localizing it (drill-down), and diagnosing it (detail).

Essential Dashboard Panels

The global overview row contains four time-series panels showing the metrics that capture CDN health at a glance: total requests per second (with day-over-day comparison), global cache hit ratio, error rate (5xx responses as percentage of total), and p95 latency. These four metrics enable an operator to assess CDN health within seconds of looking at the dashboard.

The geographic drill-down row uses a world map visualization overlaid with per-PoP metrics. Color encodes health status (green for normal, yellow for degraded, red for critical), and clicking a PoP reveals its metrics in the detail panels below. This spatial view quickly identifies regional issues — a cluster of yellow/red PoPs in a specific area suggests a network problem affecting that region rather than a CDN-wide issue.

The detail row provides three panels for deep investigation: a cache status breakdown (HIT/MISS/STALE/EXPIRED over time) that reveals cache hit ratio trends, a latency heatmap that shows the full distribution shape rather than just percentile summaries, and a top-N panel listing the most-requested URLs with their individual metrics. These detail panels answer the “why” questions that the overview panels raise — crucial context for CDN troubleshooting.

Scaling Considerations

A CDN serving 1 billion requests per day generates approximately 500 GB of raw access logs daily. After stream processing aggregation (reducing individual requests to dimensional summaries per minute), ClickHouse stores roughly 50-80 GB per day with LZ4 compression. Materialized hourly views add approximately 5% overhead while enabling sub-second queries over months of data.

The pipeline scales horizontally at every stage. Kafka partitions map to Flink parallelism, so adding edge PoPs or increasing traffic requires adding partitions and Flink task slots proportionally. ClickHouse supports distributed tables across a cluster of shards, each handling a subset of the data. Grafana queries fan out across shards and aggregate results, maintaining query performance as data volume grows.

Cost management centers on two decisions: retention policy and sampling strategy. Full-fidelity minute-level data for 30 days, hourly rollups for 12 months, and daily rollups for 3 years provides a practical tiered retention policy. For exploratory analytics (which URLs are most popular, what is the browser distribution), 1-10% sampling during stream processing reduces storage by 10-100x while providing statistically valid distributions. Core SLI metrics always use full-fidelity data to ensure accurate SLO calculations, while exploratory queries tolerate the slight inaccuracy of sampling in exchange for significant cost savings.