Custom Metrics Collection Framework for CDN: Building Bespoke Observability

Back in late 2019 we had a major streaming customer complain about stuttering playback during a live sports final, yet every dashboard we had said the CDN was healthy — 98% cache hit ratio, origin load flat, no 5xx spike. It wasn't until we dug through raw logs hours later that we found a 12% rebuffer rate isolated to a single ABR rendition and one PoP in Madrid. That incident killed our faith in vendor-default metrics and kicked off a two-year effort to build our own bespoke monitoring stack.

Why Off-the-Shelf CDN Observability Falls Short

Most CDN providers give you the same dozen graphs: requests per second, bandwidth, cache hit/miss, status codes, maybe edge CPU. That's fine for capacity planning, but it tells you almost nothing about user experience or business logic. If you're serving video, you don't just care about hit ratio — you care about time-to-first-byte segmented by cache state and codec, or rebuffer ratio per player version. If you're an e-commerce platform, you care about cache bypass rate for personalized API calls or the latency penalty of your edge workers. Standard tools can't answer those questions because they weren't designed with your traffic shape in mind.

I've made the mistake of trying to force custom use cases into generic platforms. At my last gig we tried to pipe every edge log into Datadog and build custom metrics there. It worked for a week. Then the bill arrived, our cardinality exploded, and query latency went through the roof. We were creating a new metric tag for every URL path and every edge node, and Datadog's custom metrics pricing punished us for it. We also couldn't get sub-second granularity because the logs were batched and shipped every 30 seconds. We needed a system where metric collection happened right at the edge, before we ever shipped data off-box.

That's the core idea behind a proper custom metrics framework: you don't collect more data, you collect the right data, and you aggregate it where it's cheapest. Instead of relying on post-facto log parsing, you define the signals you need and instrument the edge itself to emit them. It's more work up front, but it gives you the CDN Real-Time Log Streaming Architecture: From Edge Events to Analytics in Seconds you need to react in seconds rather than hours. Our rule now is simple: if a metric can't be tied to a user-visible symptom or a concrete autoscaling decision, we don't collect it.

// VCL / Varnish-style edge instrumentation for bespoke signals
// Emit Server-Timing and structured log fields per HTTP Working Group Specifications
sub vcl_deliver {
    // Calculate TTFB split by cache state - critical for CDN observability
    set resp.http.Server-Timing = "cdn-cache;desc=" + obj.hits + ", edge;dur=" + time.to_first_byte;

    // Custom log line with low-cardinality dimensions only
    std.log("custom_metric:ttfb_ms=" + time.to_first_byte +
            " cache=" + resp.http.X-Cache +
            " codec=" + req.http.X-Playback-Codec +
            " pop=" + server.datacenter);
    
    // Don't log high-cardinality fields like URL or user-id here
    // Those belong in sampled traces, not metrics
}

Architecting an Edge-Native Metric Collection Pipeline

There are two bad ways to build this and one that actually scales. The first bad way is to log every request to disk and have a sidecar tail the file — you'll run out of IOPS and lose data the second traffic spikes. The second is to send a UDP packet or HTTP POST per request to a central collector — you'll DDoS yourself. What works is in-worker, in-memory aggregation with periodic flushes. We run a tiny Rust aggregator directly inside our edge runtime (it was Lua for years, but we rewrote it) that holds histograms, counters, and gauges in shared memory.

Each worker thread updates local structures with zero allocations. Every 5 seconds, a background timer flushes a pre-aggregated snapshot via OTLP over a persistent mTLS connection to a regional aggregator. That aggregator deduplicates, adds PoP and cluster labels, and forwards to our Mimir/Prometheus backend. We don't use StatsD — its text protocol and lack of native histograms cost us too much CPU at 200k rps per node. Our current setup adds about 0.3% CPU and under 20MB RAM per edge instance, even while tracking 40+ custom histograms. The trick is using exponential histograms and pre-bucketing values so you aren't storing raw samples.

You'll still need logs for debugging, but metrics and logs should have separate paths. We've learned to keep our CDN Log Aggregation Architecture: Collecting Billions of Events Without Dropping Data decoupled from our metrics path so a log pipeline stall doesn't blind us. And we aggressively apply CDN Log Sampling Strategies: Reducing Storage Costs Without Losing Signal to traces and raw logs, while keeping metrics at 100% coverage because they're already aggregated. If you want to see how we implement the flush, the pattern below is close to what we run in production. For header semantics and timing we follow the HTTP Working Group Specifications and for client-side correlation we align with guidance from web.dev Articles on Core Web Vitals and Server-Timing.

# Edge aggregator - flushes pre-bucketed histograms every 5s
# Config for OpenTelemetry Collector regional tier
processors:
  memory_limiter:
    limit_mib: 512
  cumulativetodelta:
    include:
      metrics: ["edge.ttfb", "edge.compute.duration"]

exporters:
  prometheusremotewrite:
    endpoint: "https://mimir.internal/api/v1/push"
    headers:
      X-Scope-OrgID: "cdn-prod"

connectors:
  # Aggregate histograms at PoP before central write
  histogram_aggregator:
    buckets: [5, 10, 25, 50, 100, 250, 500, 1000, 2500] # ms
   Cardinality_limit: 15000

service:
  pipelines:
    metrics/edge:
      receivers: [otlp]
      processors: [memory_limiter, cumulativetodelta]
      exporters: [histogram_aggregator]

What We Actually Measure (And What We Stopped Measuring)

Early on we measured everything. We had a metric for every header, every cookie, every edge function. Our cardinality hit two million series in a day and our Prometheus compactor fell over. Now we're ruthless. Every new metric has to pass a review: what's the alert it drives, who's on call for it, and what's its cardinality budget? We've found that five or six well-chosen, low-cardinality histograms tell you more than fifty naive counters.

We bucket aggressively and we never use unbounded labels. So instead of `http_path`, we use `route_template` like `/api/product/:id`. Instead of `user_id`, we use `auth_state=anonymous|authenticated`. Instead of `url`, we use `content_type` and `cache_status`. This keeps series count predictable even when traffic patterns shift. We also drop any dimension that doesn't change alerting logic — for example, we stopped splitting TTFB by TLS version after we realized it wasn't actionable, and we kept the split by cache state and by PoP because those directly map to runbooks.

The table below is a simplified version of our current production set. If you're starting from scratch, these are the signals that have given us the highest return on toil for metric collection without breaking our storage budget.

Putting It Together: A Production-Ready Edge Collector

We learned the hard way that you don't want a heavy agent on every edge node. Our first iteration ran a full Telegraf + Prometheus sidecar per PoP and it ate 18% of CPU at peak — completely unacceptable when you're pushing 85k req/s per box. What we run now is brutally simple: a Lua shared dictionary inside OpenResty/Nginx for in-worker aggregation, and a single Vector sidecar per machine that scrapes, batches, and ships. No per-request network calls. Metrics are incremented in-memory via ngx.shared.metrics:incr(), aggregated in 1-second buckets, and flushed every 5 seconds. If Vector restarts, you lose at most 5 seconds of data, not the whole worker's state.

The trick is deciding what to aggregate at the edge versus centrally. Counters and histograms? Aggregate locally. We keep a sliding window of 60 buckets for request duration, cache hit ratio, and origin fetch time, then emit pre-computed quantiles (p50, p95, p99) plus the raw histogram buckets for ClickHouse. High-cardinality dimensions like url_path never leave the edge — we normalize them to route patterns (/product/{id}) using a radix tree compiled from our OpenAPI spec. That single change dropped our series cardinality from 2.3 million to ~42k per PoP. If you're still logging raw paths into Prometheus labels, you're going to have a bad time. For understanding navigation flows, we actually separate that concern entirely — see our approach to Clickstream Analysis at CDN Edge: Understanding User Navigation Patterns where we handle raw event streams differently from operational metrics.

Shipping is where most bespoke systems fall over. Don't push directly to your long-term store from the edge. We push to a regional aggregator tier (3 nodes per region behind Anycast) over compressed gRPC, which deduplicates, validates timestamps, and writes to ClickHouse for analytics and Prometheus remote-write for alerting. Vector's disk buffer is non-negotiable — we allocate 10GB per edge for when a regional aggregator goes dark. During last month's Frankfurt fiber cut, we buffered 47 minutes of metrics and replayed without loss. Here's the stripped-down Vector + Lua pattern we've been running in production for 14 months:

-- nginx.conf: init + log phase
-- init_by_lua_block { metrics = ngx.shared.metrics } -- 10MB shm zone

-- log_by_lua_block {
--   local key = "req:status:" .. ngx.var.status .. ":cache:" .. ngx.var.upstream_cache_status
--   metrics:incr(key, 1)
--   metrics:incr("latency:" .. matched_route, ngx.var.request_time * 1000)
-- }

# vector.toml on the same host
[sources.edge_metrics]
type = "exec"
command = ["lua", "/etc/vector/scrape_shm.lua"]
interval_secs = 5

[transforms.aggregate]
type = "lua"
inputs = ["edge_metrics"]
# normalize, drop healthcheck routes, cap cardinality
source = '''
  if payload.path == "/health" then return end
  payload.route = normalize_route(payload.path)
'''

[sinks.regional_aggregator]
type = "vector"
inputs = ["aggregate"]
address = "aggregator.region.internal:6000"
compression = "gzip"
buffer.type = "disk"
buffer.max_size = 10737418240  # 10GB
retry.max_duration_secs = 3600

Advanced Patterns: Taming Cardinality and Chasing Ghost Metrics

If your custom framework doesn't have a cardinality circuit breaker, it isn't production-ready. We hit 180k new series in 20 minutes when a bad deploy started emitting user_id as a label — Prometheus OOM'd, and our alerting went blind for 11 minutes. Now every metric passes through a hard limiter: max 50k series per PoP, max 12 labels per metric, and any label with >500 unique values in a 5-minute window gets automatically hashed into a __dropped_high_card__ bucket and logged. We also use adaptive sampling for success traffic. We keep 100% of 4xx/5xx and origin errors, 10% of cache MISS, and just 1% of cache HIT 2xx for detailed tracing. That cuts our ingest from ~1.8M samples/sec to ~210k/sec per region without losing the signals that actually page us. For deeper behavioral analysis, we don't overload the metrics pipeline — that data goes through our User Journey Analytics from CDN Logs: Mapping the Path to Conversion pipeline instead.

The nastiest bugs aren't missing metrics, they're ghost metrics that look correct but aren't. Clock skew is the classic one. When we added edge-measured TTFB, we saw a consistent 12-15ms negative drift on 4% of nodes — turned out their chrony was syncing to a local switch with a bad oscillator. We now emit a edge_clock_delta_ms gauge by comparing edge time against our aggregator's receive time, and we alert if |delta| > 5ms for more than 2 minutes. Another ghost: worker restarts silently resetting lua_shared_dict. Your graphs will show a perfectly normal dip that hides a 30-second blip of uncounted 502s. Fix it by persisting a monotonic counter to a tmpfs file on flush and rehydrating on init_worker_by_lua.

Finally, don't try to replicate client-side attribution in your metrics pipeline. You'll double-count and drive yourself crazy reconciling numbers with Marketing. Edge observability should tell you *what* the CDN did, not *why* the user converted. We keep a strict boundary: operational metrics (hit ratio, origin latency, error budget burn) stay in ClickHouse/Prometheus, while business events are handled via first-party edge tracking that never touches the browser. If you need that last piece, our setup for Conversion Attribution from CDN Edge: First-Party Tracking Without Client-Side Scripts explains how we emit attributable events without polluting the observability stream or getting clipped by ITP. That separation alone saved us 22% on Prometheus storage and ended the weekly "why don't these dashboards match?" meeting.

Frequently Asked Questions