CDN Access Log Analysis: Parsing, Querying, and Visualizing Edge Logs

CDN access logs are the ground truth of what happens at the edge. Every client request, cache decision, TLS handshake, and origin fetch leaves a trace. When parsed and queried correctly, these logs reveal cache efficiency, origin offload, client performance, and early signs of abuse that real-time dashboards often miss. This guide shows how to build a reliable pipeline from raw edge logs to actionable insights.

Unlike synthetic monitoring, edge logs capture 100% of traffic. Understanding them is foundational to how a CDN works in production, and critical for any serious CDN monitoring and observability strategy.

1. What Edge Logs Actually Contain

Most providers (Cloudflare, Fastly, Akamai, CloudFront) emit logs in JSON, CSV, or custom W3C-extended formats. Despite syntax differences, the semantics converge around a core event model:

Advanced fields like tls_version, ja3_fingerprint, http_version, edge_colo, and origin_response_time are invaluable for security and performance deep dives. If your provider supports custom log fields, enable cache_key and surrogate_key — they explain why a request missed.

2. Log Delivery Architecture

Edge logs are high-volume and bursty. A single busy property can generate 50K–500K events per second. Never poll an API for logs in production; use push-based delivery.

Partition your storage by year/month/day/hour and optionally by colo. This reduces scanned data by 95%+ for time-bound queries. Convert JSON to Parquet or ORC with Snappy/ZSTD compression; expect 8–12x compression and 20x faster scans.

3. Parsing: From Raw Lines to Typed Events

Parsing fails silently when you treat logs as strings. Define a strict schema and reject malformed rows to a dead-letter prefix.

Common pitfalls

• Timestamp drift: Edge time is UTC; origin time may differ. Always use edge_timestamp as canonical.
• Escaped URIs: URIs contain commas, quotes, and newlines. Use a proper CSV/JSON parser, never split(',').
• Cache status nuance: HIT vs REVALIDATED vs STALE have different billing and performance implications. Map provider-specific values to a canonical enum.
• Sampling: Some CDNs sample logs at 1:10 or 1:100 under load. Check sample_rate field and extrapolate.

Example: robust Python parser for CloudFront/JSON-style logs with validation:

import json, gzip
from datetime import datetime

def parse_edge_log(path):
    # Handles gzipped JSON lines, validates schema
    with gzip.open(path, 'rt') as f:
        for line in f:
            rec = json.loads(line)
            try:
                yield {
                    "ts": datetime.fromisoformat(rec["timestamp"]),
                    "colo": rec["edgeLocation"],
                    "status": int(rec["sc-status"]),
                    "cache": rec["x-edge-result-type"], # Hit/Miss/RefreshHit
                    "ttfb_ms": float(rec.get("time-to-first-byte", 0)),
                    "uri": rec["cs-uri-stem"] + rec.get("cs-uri-query", ""),
                    "bytes": int(rec["sc-bytes"]),
                }
            except (KeyError, ValueError) as e:
                log_dead_letter(line, e)  # never drop silently

For high throughput, use Vector, Fluent Bit, or Logstash with a JSON codec and schema registry. Validate with JSON Schema or Protobuf to catch provider format changes before they break dashboards.

4. Querying at Scale

Once normalized, the real value comes from analytical queries. Three patterns cover 90% of operational questions.

A. Cache efficiency and origin offload

-- Athena / Presto / ClickHouse SQL: hourly hit ratio by PoP
SELECT
  date_trunc('hour', timestamp) AS hr,
  colo,
  count(*) AS reqs,
  sum(CASE WHEN cache_status IN ('HIT','STALE','REVALIDATED') THEN 1 ELSE 0 END) * 1.0 / count(*) AS hit_ratio,
  sum(bytes_out) / 1024/1024/1024 AS gb_served
FROM cdn_logs
WHERE timestamp BETWEEN now() - interval '7 day' AND now()
GROUP BY 1,2
ORDER BY hr DESC;

This query directly quantifies the performance and security benefits of a CDN — high hit ratios mean lower origin load and faster TTFB. A sudden drop in hit ratio often signals cache fragmentation or an accidental Cache-Control: no-store deployment.

B. Error and origin health

-- Find 5xx spikes correlated with origin vs edge
SELECT uri, cache_status, status, count(*) AS c,
       approx_percentile(ttfb_ms, 0.95) AS p95_ttfb
FROM cdn_logs
WHERE status >= 500
GROUP BY 1,2,3
HAVING c > 100
ORDER BY c DESC LIMIT 20;

C. Security and bot analysis

Edge logs are your earliest DDoS protection signal. Query for cardinality explosions:

-- Detect cache-busting bot: high unique query strings per IP
SELECT client_ip, count(DISTINCT uri) AS uniq_urls, count(*) AS reqs
FROM cdn_logs
WHERE cache_status = 'MISS' AND timestamp > now() - interval '1 hour'
GROUP BY 1
HAVING uniq_urls > 500
ORDER BY uniq_urls DESC;

For ecommerce workloads, join logs with product IDs to see cache behavior per SKU — essential context from our CDN for ecommerce guide — and prioritize image optimization for the most-requested assets.

5. Visualizing Edge Data Effectively

Visualization should answer operational questions in under 10 seconds. Avoid vanity metrics.

• Cache performance board: Hit ratio, origin offload (GB), and p50/p95 TTFB by PoP and content type. Use stacked area for HIT vs MISS over time.
• Error triage board: 4xx/5xx rate, top URIs by error count, and origin response time heatmap. Link each panel to a pre-filtered log query via Grafana data links.
• Security board: Requests per IP, JA3 diversity, and WAF action rate. Add anomaly detection on request rate per /24.

Prefer Grafana with ClickHouse or Athena plugins, or Superset for ad-hoc exploration. For executive views, aggregate to hourly and visualize bandwidth savings as dollars saved on origin egress.

6. Retention, Cost, and Governance

At 10 TB/day, raw JSON on S3 Standard becomes expensive. Apply lifecycle rules: Standard for 7 days, then Intelligent-Tiering or Glacier Instant for 90 days. Store aggregates in ClickHouse or BigQuery for fast long-term trends. Always hash or truncate client_ip after 30 days for privacy compliance, and drop cookie and authorization headers at ingestion — never persist them.

Instrument your pipeline itself: monitor log delivery lag (edge time vs warehouse ingestion time), dead-letter rate, and partition completeness. A silent gap in logs is worse than no logs at all.

When done right, CDN access log analysis turns opaque edge behavior into a tunable system. You can prove cache efficiency, isolate a slow PoP, or catch a botnet before it hits your origin — all from the same dataset. Start with a typed schema, partition wisely, and build dashboards that link directly back to the underlying events.