CDN Observability Correlation: Connecting Logs, Metrics, and Traces Across Edges

Last November I got paged at 2:13 AM because our checkout API's error rate jumped to 4% in one region while the global dashboard stayed stubbornly green. Metrics said we were fine, logs showed a flood of 502s from a single POP in Frankfurt, and traces — what few we had — didn't connect either signal. That night forced us to stop treating logs, metrics, and traces as separate tools and build a real observability correlation layer that actually works at the edge.

Why Edge Aggregates Hide the Failures That Matter

I've run CDN platforms for over a decade, and the same trap catches every team: edge metrics are too aggregated to be useful when you need them most. A global p95 latency of 85ms looks healthy until you realize it’s averaging 250 POPs, three cache tiers, and millions of requests per second. We had a case where our Anycast edge in São Paulo was serving stale content for six minutes after a purge, but our 60-second rollup hid it completely — the 1% of requests hitting that POP were drowned out by the other 99% hitting cache. Without distributed tracing context attached to every edge event, you can't isolate whether that latency spike is a client network issue, a cache revalidation storm, or an origin slowdown.

That's where CDN observability breaks down compared to traditional backend monitoring. In a central data center, you can assume a request flows linearly through your services. At the edge, a single user request might hit a shield POP, collapse with 200 other requests, miss cache, spawn a sub-request to origin, and get served from a completely different thread. If your metrics aren't joined to a trace ID and your logs don't carry the same correlation ID, you're left guessing. We tried for months to correlate by timestamp and URL — it doesn't work. Clock skew between POPs alone will kill you, and URL cardinality makes joins impossibly expensive.

What finally worked was forcing a correlation fabric at the very first byte. Every request that hits our edge — even a cache HIT that never reaches origin — gets a W3C Trace Context header if it doesn't already have one. The spec from the HTTP Working Group Specifications defines traceparent exactly for this, and we follow it strictly so downstream services don't drop the context. Here's the simplified Worker logic we run on every POP before cache lookup, not after:

// Edge correlation injection - runs BEFORE cache lookup
export default {
  async fetch(request, env, ctx) {
    let traceparent = request.headers.get('traceparent');
    let tracestate = request.headers.get('tracestate') || '';

    // Generate if ingress has no trace context (most browser requests don't)
    if (!traceparent) {
      const traceId = Array.from(crypto.getRandomValues(new Uint8Array(16)))
        .map(b => b.toString(16).padStart(2, '0')).join('');
      const spanId = Array.from(crypto.getRandomValues(new Uint8Array(8)))
        .map(b => b.toString(16).padStart(2, '0')).join('');
      traceparent = `00-${traceId}-${spanId}-01`;
      tracestate = `cdn=pop:${env.POP_ID},cache:miss`;
    }

    // Inject into request for origin and sub-requests
    const correlatedRequest = new Request(request, {
      headers: { ...Object.fromEntries(request.headers), traceparent, tracestate }
    });

    // Attach to logger and metrics context - critical for HITs
    ctx.correlation = { traceId: traceparent.split('-')[1], spanId: traceparent.split('-')[2] };
    return fetch(correlatedRequest);
  }
}

Building a Correlation Fabric That Survives Cache Hits and Collapses

Most teams inject trace IDs at the origin or at the edge *after* a cache miss. That misses 85-95% of your traffic. If you only trace origin fetches, your log-metric correlation is blind to the exact requests users actually experience. I learned this the hard way when we spent two weeks debugging a TTFB regression that our origin traces said didn't exist — because 92% of the slow requests were served directly from edge RAM and never generated a trace.

Your correlation IDs have to survive three edge-specific nightmares: cache hits, request collapsing (also called coalescing), and tiered shielding. A cache HIT should still emit a log line and a histogram observation with the original trace ID, even though it generates no new span. A collapsed request — where 50 clients wait on one origin fetch — needs to produce 50 log entries with distinct trace IDs but a shared parent span for the collapsed origin fetch. We handle this by storing the traceparent in a shared memory key alongside the cache entry metadata, not in the cached object itself. That way a HIT can re-emit the original trace context without re-validating.

This is also where your log pipeline design makes or breaks you. You can't correlate after the fact if your logs are sampled differently than your traces. We align sampling decisions at the edge using the trace ID's head-based sampling bit (the last two characters of the flags field). If you're still designing your pipeline, get your CDN Real-Time Log Streaming Architecture: From Edge Events to Analytics in Seconds right first — late logs mean broken exemplars. And for persistence, our CDN Log Aggregation Architecture: Collecting Billions of Events Without Dropping Data keeps the trace_id and span_id as indexed fields in ClickHouse, not just as a text blob, so we can join in milliseconds:

{
  "timestamp": "2024-11-15T02:13:42.118Z",
  "pop": "fra-07",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "parent_span_id": "c8c7b3f0a1e2d4a5b",
  "cache_status": "HIT",
  "cache_tier": "edge-ram",
  "request_collapsed": false,
  "tracestate": "cdn=pop:fra-07,cache:hit",
  "http_status": 200,
  "ttfb_ms": 12,
  "user_agent": "Mozilla/5.0...",
  "exemplar": true
}

Joining Logs, Metrics, and Traces Without Exploding Cardinality

Here's the mistake I see senior teams make over and over: they try to add trace_id as a Prometheus label. Don't. Your metrics system will fall over in an hour. High-cardinality correlation has to use exemplars, not labels. We attach exemplars to our histogram buckets — specifically edge_request_duration_seconds and origin_fetch_duration_seconds — so a metric query can jump directly to two or three representative traces without storing millions of label combinations. OpenTelemetry's exemplar support and Prometheus 2.40+ make this practical, but only if your edge daemons actually emit them.

The second mistake is mismatched sampling. If you sample logs at 1:100 and traces at 1:1000 independently, your join rate is effectively 1:100,000 — useless for debugging tail latency. You need consistent head-based sampling or at least tail-based sampling that preserves errors. For most CDN workloads, I recommend keeping 100% of 5xx and cache MISS logs while sampling HITs aggressively, but *always* preserving the trace for any request that exceeds your SLO. The web.dev Articles on Core Web Vitals show how much TTFB variance matters for LCP — you can't afford to sample away your slowest 1%. We detail a cost-effective approach in CDN Log Sampling Strategies: Reducing Storage Costs Without Losing Signal that keeps exemplar joins intact.

Once you've got exemplars and aligned sampling, the workflow flips. Instead of hunting through logs for a timestamp, you start from a metric anomaly — say, p99 TTFB in gru spiking — click the exemplar, get the trace, and pivot to logs with that exact trace_id filtered to that POP. It cuts our mean time to root cause from 45 minutes to under 5. To keep it sustainable, you need strict guardrails on what you correlate:

// Cloudflare / Fastly Compute@Edge style worker - inject & propagate
export default {
  async fetch(request, env, ctx) {
    const traceId = request.headers.get('traceparent') 
      || `00-${crypto.randomUUID().replace(/-/g,'')}-${crypto.randomUUID().slice(0,16)}-01`;
    
    // Low-cardinality correlation ID for metrics/logs join
    const corrId = `${traceId.split('-')[1].slice(0,8)}:${request.cf.colo}:${Date.now()}`;

    const res = await fetch(request, {
      headers: { ...request.headers, 'traceparent': traceId, 'x-corr-id': corrId }
    });

    // Ensure hit still emits log with trace context
    ctx.waitUntil(logToCollector({
      timestamp: Date.now(),
      trace_id: traceId.split('-')[1],
      pop: request.cf.colo,
      cache_status: res.headers.get('cf-cache-status') || res.headers.get('x-cache'),
      ttfb_ms: Date.now() - request.cf.requestStart,
      status: res.status,
      corr_id: corrId
    }));

    const newRes = new Response(res.body, res);
    newRes.headers.set('traceparent', traceId);
    newRes.headers.set('server-timing', `edge;dur=${Date.now() - request.cf.requestStart}`);
    return newRes;
  }
}