API Metering at CDN Edge: Real-Time Usage Tracking for SaaS Billing

Last spring a SaaS customer called us at 2 a.m. because their biggest tenant had blown past their API quota three days before the end of the billing cycle and nobody noticed. The origin logs said 11 million requests, Stripe said 8 million, and the customer was left eating the difference. That gap wasn't a bug in their billing code — it was the delay between edge and origin. If you're metering APIs at your origin or in a nightly batch job, you're billing on stale data, and at scale that'll cost you real money and trust.

Why Centralized Metering Can't Keep Up With Edge Traffic

For years the standard pattern was simple: let the CDN cache what it can, forward the rest to origin, and count everything there. It works fine when you're serving pages. It falls apart the moment you start charging per API call, per function execution, or per gigabyte of transformed images. I ran a platform that did exactly this for a multi-tenant analytics API back in 2018. We had NGINX logs shipping to S3, parsed by Spark every hour, pushed to Postgres. On paper the pipeline was solid. In practice, we were always 45 to 90 minutes behind, we lost about 0.8% of log lines to rotation and delivery failures, and any tenant bursting over their rate limit had already consumed thousands of free calls before we could cut them off.

The problem is physics as much as architecture. Your CDN might have 300+ PoPs, but your origin is in two or three regions. Every request that has to round-trip for a metering decision adds latency you can't afford to put on a billing path, and every log that has to be shipped centrally is a chance to drop data. When you're doing CDN Multi-Tenant Architecture: Isolating Customer Traffic While Sharing Infrastructure, that loss doesn't distribute evenly — one noisy tenant can drown out counters for everyone else if you're using a shared log stream. Real-time SaaS billing demands the counter lives where the request lands, not where it ends up.

Edge metering flips that model. Instead of observing traffic after the fact, you count and classify it inside the PoP as it happens, then replicate the counts. You'll still need a central system of record, but your edge becomes the source of truth for quantity and timing. We moved to per-PoP atomic increments with periodic flush to durable storage, and our discrepancy against origin dropped from ~1.2% to under 0.05% even during a 4x traffic spike that took down our central collector for 19 minutes. The edge kept counting.

// Cloudflare Workers / Fastly Compute@Edge pattern: atomic edge counter
// Durable Object / Edge KV used for per-tenant aggregation window (60s)
export async function handleRequest(request, env) {
  const tenantId = request.headers.get('x-tenant-id');
  const apiKeyTier = await getTierFromCache(request.headers.get('x-api-key'));
  
  // Increment happens inline, no origin round-trip
  const counterKey = `usage:${tenantId}:${new Date().toISOString().slice(0,16)}`;
  
  // Fire-and-forget increment with retry queue - don't block response
  env.METERING_DO.get(env.METERING_DO.idFromName(tenantId))
    .incrementAndCheckQuota({
      key: counterKey,
      tenantId,
      tier: apiKeyTier,
      endpoint: new URL(request.url).pathname,
      bytesOut: 0 // updated in waitUntil after response
    });

  // Continue to fetch/cache logic immediately
  return fetch(request);
}

// Inside Durable Object - sliding window check
async incrementAndCheckQuota(event) {
  const current = (await this.storage.get(event.key) || 0) + 1;
  await this.storage.put(event.key, current);
  // async replicate to central ClickHouse / BigQuery via waitUntil
  this.state.waitUntil(this.flushToCentral(event, current));
  return current;
}

Building a Lossless Edge Pipeline: Counters, Queues, and Flush Guarantees

You don't need perfect real-time consistency across all PoPs to bill accurately, but you do need durability and ordering guarantees for the increments. The worst approach I've seen — and we've all shipped it — is logging to local disk or stdout at the edge and hoping a collector scrapes it. PoPs get evacuated, processes restart, and you lose a window. What you want is an in-memory counter backed by a replicated store at the edge, with a bounded flush interval. I've had good results with a 5-second in-memory aggregation and a 10 to 60-second flush to central storage. Shorter than 5 seconds and you're hammering your global database; longer than 60 and your quota enforcement and real-time dashboards feel laggy.

The reference architecture that hasn't failed me yet looks like this: a lightweight WASM or VCL snippet does classification (tenant, endpoint, status code, billable or not), increments a local counter, and pushes an event to a durable edge queue. That queue drains via reliable waitUntil or background task to your analytics store — we use ClickHouse for this — and also updates an edge KV for immediate quota checks. If you're implementing CDN Tenant Isolation Patterns: Cache Separation, Rate Limits, and Access Controls, you can piggyback on that same KV for per-tenant token buckets without adding another lookup. The key detail is idempotency. Every increment needs a request ID that survives retries, otherwise you'll double-count on PoP failover. The HTTP Working Group Specifications around idempotency keys and retry behavior are useful reading here, even if you're not exposing them to clients directly.

Don't bill on cache HIT vs. MISS alone either. For usage tracking, you care about what was served and what was billable per your contract. A 304 Not Modified or a 206 Partial Content might be free for one product and billable for another. Same with GraphQL — one POST to /graphql could be one billable operation or ten, depending on how you define it. Your edge classifier needs to understand that. We parse the GraphQL AST at the edge for one customer and count selections, not HTTP requests, because that's what their pricing page promises. It's a bit more CPU per request, but it's the only way the invoice matches what the developer expects to see.

# Fastly VCL snippet - classification before counting
# Distinguish billable API calls from cache hits, health checks, and free tier

sub vcl_recv {
  # Normalize tenant from custom domain or header - critical for white-label
  if (req.http.Host ~ "^(.*)\.customer-api\.com$") {
    set req.http.X-Tenant-ID = re.group.1;
  }
  
  # Non-billable paths
  if (req.url ~ "^/healthz|^/favicon.ico") {
    set req.http.X-Billable = "0";
    return(lookup);
  }
}

sub vcl_deliver {
  # Only count 2xx and 429s as billable attempts per our contract
  if (resp.status >= 200 && resp.status < 300) {
    set req.http.X-Billable = "1";
  } else if (resp.status == 429) {
    set req.http.X-Billable = "1"; # rate-limited calls still count against quota
  } else {
    set req.http.X-Billable = "0";
  }

  # Emit structured log for edge queue - includes bytes, not just hits
  log "syslog serviceId metering :: "
    "tenant=" req.http.X-Tenant-ID " "
    "endpoint=" req.url.path " "
    "billable=" req.http.X-Billable " "
    "cache=" fastly_info.state " "
    "bytes=" resp.http.Content-Length " "
    "req_id=" req.http.X-Request-ID;
}

What to Actually Meter (and What It Costs to Meter It)

Not every signal is worth counting at the edge. Request count is cheap — an integer increment. Egress bytes, function duration, or AI tokens are more expensive because you only know the value after you generate the response. I've learned to split metering into two phases: synchronous (before the response) and asynchronous (after). Tenant ID, API key tier, endpoint pattern, and method are synchronous and belong in the hot path. Response size, compute milliseconds, and cache status are asynchronous and should be recorded in vcl_deliver or waitUntil after you've sent headers. If you block the response waiting to measure bytes, you'll add 5-15ms you didn't need to.

For SaaS products that offer White-Label CDN Delivery: Custom Branding for SaaS Customer Domains, tenant resolution is its own headache. You can't rely on a single header when every customer brings their own domain and custom auth scheme. We've standardized on resolving tenant in three steps at the edge: SNI/host map, then API key prefix lookup in edge KV (sub-millisecond), then JWT claim if needed. That order matters for cost. A KV lookup is ~1-2ms, a JWT verify is 4-7ms in WASM. Do the cheap check first, and cache the mapping for 60 seconds. The web.dev Articles on edge caching strategies helped us tune those TTLs without creating stale entitlement issues when a tenant upgrades mid-cycle.

Here's how the trade-offs break down in practice when we benchmarked different metering granularities on a 42-PoP deployment handling ~1.8M RPS peak. These aren't marketing numbers — they're from our own load tests last quarter with Compute@Edge and Workers.

Putting Metering Into Production Without Adding Latency

Here's where most teams get tripped up: they build a brilliant metering pipeline on paper, then watch p95 latency jump 40-60ms because they're doing a synchronous fetch to Redis or Postgres on every request. You can't block the hot path. The pattern that actually works at scale is fire-and-forget with local aggregation — increment an in-memory counter in the worker, batch it, and flush asynchronously. We've run this on Cloudflare Workers and Fastly Compute and consistently keep the metering overhead under 3ms at p99, even at 85k req/s per PoP.

The trick is to keep your edge state ephemeral and your flush interval short. Don't try to maintain perfect per-customer counters in Durable Objects or edge KV for every request — that's expensive and slow. Instead, use the worker's global memory to bucket counts by customer_id + endpoint_tier + status_class for a 5-10 second window. On a timer or when the bucket hits 1,000 events, you push a single aggregated message to a queue. That drops your queue writes by 99.7% versus per-request logging. If you're running per-customer vanity domains, you'll want to resolve the customer from the hostname at the edge too, which is exactly what we do with Custom Domain CDN for SaaS: Automated SSL and DNS for Customer Vanity Domains — the TLS SNI mapping gives you the tenant ID without a lookup.

For the transport, don't send directly to your billing DB. Send to a durable queue first. On Cloudflare that's Queues or Vector + Kafka; on Fastly/AWS it's Kinesis or Pub/Sub. The queue is your shock absorber when your ClickHouse or BigQuery sink lags. We've seen ingest lag spike to 4 minutes during ClickHouse merges — without a queue, you'd drop ~14 million events per minute at scale. With it, you just replay. A minimal worker looks like this:

// Cloudflare Worker - edge metering with batched flush
const BUCKETS = new Map();
const FLUSH_INTERVAL_MS = 5000;
const FLUSH_THRESHOLD = 1000;

function record(customerId, tier, bytes, isCacheHit) {
  const key = `${customerId}:${tier}`;
  let bucket = BUCKETS.get(key);
  if (!bucket) {
    bucket = { count: 0, bytes: 0, hits: 0, tier, customerId };
    BUCKETS.set(key, bucket);
  }
  bucket.count++;
  bucket.bytes += bytes;
  if (isCacheHit) bucket.hits++;

  if (bucket.count >= FLUSH_THRESHOLD) flush(key);
}

async function flush(key) {
  const bucket = BUCKETS.get(key);
  if (!bucket || bucket.count === 0) return;
  BUCKETS.delete(key);
  // Non-blocking send — don't await in request path
  ctx.waitUntil(
    fetch('https://queue-ingest.your-cdn.net/batch', {
      method: 'POST',
      body: JSON.stringify({
        ...bucket,
        ts: Date.now(),
        pop: POP_ID,
        edge_ms: performance.now() - start
      }),
      headers: { 'content-type': 'application/json' }
    }).catch(err => console.error('meter flush failed', err))
  );
}

// Flush every 5s regardless of threshold
setInterval(() => { for (const k of BUCKETS.keys()) flush(k); }, FLUSH_INTERVAL_MS);

export default {
  async fetch(req, env, ctx) {
    const start = performance.now();
    const customerId = req.headers.get('x-tenant-id') || resolveFromHostname(req);
    const res = await fetch(req);
    // Don't block response on metering
    record(customerId, getTier(req.url), parseInt(res.headers.get('content-length')||0), res.headers.get('cf-cache-status') === 'HIT');
    return res;
  }
}

Advanced Patterns: Deduplication, Backpressure, and Reconciliation

Once you're past the first 10M requests/day, the edge cases will get you. The biggest one is double-counting on retries. When your origin times out after 2 seconds, the CDN will often retry to another PoP or re-issue the request from queue consumer after a failure. If you naively count every edge invocation, you'll overbill customers by 2-4%. We've solved this by attaching a x-request-id + x-attempt header at ingress and using a 24-hour HyperLogLog or Redis Bloom filter in your aggregator to deduplicate before ClickHouse insert. It costs about 1.2KB per 10k IDs and cuts billing disputes to near zero.

Backpressure is the second killer. Your billing pipeline *will* fall behind — ClickHouse will stall, Kafka will throttle, or your Stripe usage record API will rate-limit you at 100 writes/sec. Don't let edge workers block or drop. Implement a circuit breaker: if queue publish fails twice in a row, spill to local disk/R2/S3 as newline-delimited JSON and run a reconciler every minute that replays those spill files. In production we've seen this spillover handle 18-minute downstream outages with zero loss, replaying at ~45k events/sec once the sink recovers. Monitor your queue depth per PoP; if it exceeds 50k messages or age > 90 seconds, page on it.

Finally, you need a cold-start reconciliation job. Edge counters are eventually consistent, and clock skew across 300+ PoPs means your real-time dashboard will drift 0.5-1.5% from your authoritative logs. Nightly, run a batch job that re-aggregates raw CDN logs (your Logpush/S3 access logs) and compares against your real-time ClickHouse table for the same hour. Any delta > 2% should auto-correct before you push to Stripe or Orb. For teams building full analytics on top of this, tie it into CDN Usage Tracking for SaaS: Per-Customer Bandwidth and Request Analytics so your product and finance dashboards share the same source of truth. And if you're metering authenticated enterprise traffic, remember identity matters — you'll want your metering to respect tenant isolation from Enterprise SSO Integration with CDN: SAML, OIDC, and Custom Identity Providers so you don't leak usage between orgs.

Frequently Asked Questions