Clickstream Analysis at CDN Edge: Understanding User Navigation Patterns

Last November, I was on call when a retail client’s product team swore their new navigation menu had improved discovery by 22%. Their origin analytics said so. But our edge logs told a completely different story — users were bouncing between category pages in a loop we’d never seen before, and it wasn’t showing up in their Google Analytics at all. That’s the gap with clickstream analysis at the edge: you finally see what users actually do, not just what your origin manages to record.

Why the Edge Sees User Navigation Your Origin Will Never Catch

I’ve spent the last decade watching teams try to understand user navigation from origin logs and client-side tags, and they’re always working with an incomplete picture. By the time a request hits your origin, you’ve already lost 60-80% of the story. Cached assets, 304 Not Modified responses, prefetch requests, and abandoned navigations never make it there. If you’re doing serious behavior tracking, the CDN edge is the only place where you see the full sequence — every click, every back button, every rage-reload when an image is slow to paint.

We learned this the hard way with a media customer in 2021. They were trying to map content discovery paths and couldn’t figure out why their “related articles” widget had a near-zero click-through rate in their analytics. At the edge, we could see users *were* clicking, but the browser’s prefetch was firing so many speculative requests that their tag manager was throttling events. Their CDN Real-Time Log Streaming Architecture: From Edge Events to Analytics in Seconds wasn’t set up to distinguish prefetch from real navigation, so the signal was drowned out. Once we filtered on `Sec-Purpose: prefetch` and `Next.js` prefetch headers per the HTTP Working Group Specifications, the real clickstream emerged and they redesigned the widget entirely.

Edge analytics also gives you timing you can’t reconstruct later. At the origin you get a timestamp when your app server handled the request. At the edge you get the client’s actual inter-click interval, TLS handshake time, and whether the navigation was a hard reload, a back/forward cache restore, or a client-side route transition. For clickstream analysis that distinction matters — a user who hits “back” after 1.2 seconds is signaling something very different from one who navigates forward after 40 seconds. Building that fidelity in at the log line is far easier than inferring it downstream.

# Custom log format for edge clickstream - Varnish / NGINX style
# We capture what origin logs miss: navigation type, cache state, and client timing

log_format clickstream '$remote_addr - $request_id '
                      't=$msec cid=$http_x_click_id '
                      'method=$request_method uri="$request_uri" '
                      'status=$status cache=$upstream_cache_status '
                      'referer="$http_referer" nav_type="$http_sec_fetch_mode" '
                      'purpose="$http_sec_purpose" rtt=${request_time} '
                      'ua="$http_user_agent" tls_ver=$ssl_protocol';

# Example edge output:
# 203.0.113.45 - req-8f3a1b t=1701439921.412 cid=abc-123 method=GET uri="/category/shoes?sort=new" status=200 cache=HIT referer="https://shop.example.com/" nav_type="navigate" purpose="-" rtt=0.042 ua="Mozilla/5.0..." tls_ver=TLSv1.3

Stitching Clicks Into Sessions Without Relying on Broken Cookies

Anyone who’s tried to do clickstream analysis with just cookies in the last three years knows it’s a losing battle. ITP, ETP, third-party cookie blocking, and users clearing site data mean your session IDs fracture constantly. I worked on a project where 31% of sessions broke mid-journey because Safari’s 7-day cap expired while a user was comparison shopping over a weekend. We were attributing a single high-intent path to three separate users. If you’re tying behavior tracking to a `_ga` or `session_id` cookie alone, your user navigation graphs are already wrong.

At the edge we’ve moved toward a hybrid session stitching approach that doesn’t depend on persistence. You combine a short-lived edge-assigned click ID (`x-click-id`) with a deterministic fingerprint for anonymous stitching, and you use `request.headers` that are actually stable. The trick isn’t to be creepy or try to fingerprint for advertising — it’s to maintain continuity for the next 15-30 minutes so a multi-page journey stays together. We hash IP + User-Agent + TLS fingerprint + Accept-Language at the edge worker and rotate the hash daily. It’s not perfect for long-term identity, but for clickstream paths it keeps 95%+ of journeys intact even when cookies are blocked. Guidance on handling these client hints without breaking privacy is well covered in web.dev Articles.

What makes this workable is that you don’t need a central database lookup on every request. The edge worker can set and read the stitching identifier itself, and forward it as a structured log field. Your downstream pipeline then groups by that identifier instead of trying to join cookies after the fact. If you’re still funneling all raw logs to a single origin collector for this, you’ll want to read through CDN Log Aggregation Architecture: Collecting Billions of Events Without Dropping Data before your pipeline falls over at peak traffic.

// Edge Worker (Cloudflare / Fastly Compute@Edge) - session stitching for clickstreams
addEventListener('fetch', event => {
  event.respondWith(handleRequest(event));
});

async function handleRequest(event) {
  const req = event.request;
  let clickId = req.headers.get('x-click-id');
  let sessionId = req.headers.get('x-session-id');

  // Generate stable but short-lived session hash if no cookie present
  if (!sessionId) {
    const fpSource = [
      req.headers.get('user-agent') || '',
      req.headers.get('accept-language') || '',
      event.clientAddress || ''
    ].join('|');
    
    // Daily salt prevents long-term tracking, allows 30-min path stitching
    const salt = new Date().toISOString().slice(0, 10);
    const hash = await sha256(fpSource + salt);
    sessionId = hash.slice(0, 16);
  }

  // Propagate for logging and downstream clickstream assembly
  const newHeaders = new Headers(req.headers);
  newHeaders.set('x-session-id', sessionId);
  newHeaders.set('x-click-id', clickId || crypto.randomUUID());

  const res = await fetch(req, { headers: newHeaders });
  
  // Log edge-native navigation signals
  console.log(JSON.stringify({
    sessionId,
    url: req.url,
    referer: req.headers.get('referer'),
    fetchMode: req.headers.get('sec-fetch-mode'),
    fetchDest: req.headers.get('sec-fetch-dest'),
    cacheStatus: res.headers.get('x-cache')
  }));
  
  return res;
}

What We Actually Trade Off When We Sample and Ship Clickstreams

Here’s where idealism meets the monthly bill. A mid-size e-commerce site doing 200 million edge requests a day can generate 40-60 GB of uncompressed clickstream logs daily if you log every navigation event with full headers. I’ve seen teams turn on 100% logging with high-cardinality fields and then get a very uncomfortable call from finance. You don’t need every single heartbeat request to understand user navigation patterns, but you do need to be deliberate about what you drop. That’s why CDN Log Sampling Strategies: Reducing Storage Costs Without Losing Signal should be part of your design from day one, not an afterthought when storage costs triple.

For clickstream analysis specifically, we typically recommend different sampling tiers by event type. Navigational page views (`sec-fetch-mode: navigate`) and `document` destinations get sampled at 100% or at worst 50% — they’re the backbone of your path analysis. Subresource requests, beacons, and API polling can be sampled at 5-10% or even excluded from the clickstream pipeline entirely. The nuance is in bot filtering at the edge before sampling. If you sample first and then filter bots, your data skews. Filter known crawlers, prefetch, and synthetic monitoring at the edge, then sample the human navigation stream. That keeps your behavior tracking accurate while cutting volume by half or more.

The other cost isn’t storage, it’s query latency. Edge analytics tables that aren’t partitioned by session time and POP quickly become unqueryable. We partition by hour and edge POP, cluster by `sessionId`, and keep a 14-day hot window in ClickHouse for path queries. Anything older gets rolled to Parquet in S3 and queried via Athena. It’s not glamorous, but it’s the difference between a product manager getting a navigation funnel in 3 seconds versus waiting 4 minutes and giving up.

Building Clickstream Capture That Actually Survives in Production

We’ve talked theory long enough. Putting this into production at the edge is where most teams trip. You don’t want a heavy client-side SDK phoning home — that defeats the whole purpose. You want a lightweight edge function that stamps every request as it passes through, without adding more than 3-5ms of latency. At Cloudflare, Fastly, or Akamai, that’s a Worker, Compute@Edge, or EdgeWorker that runs before cache lookup. It checks for your first-party `__csid` cookie, generates one if it’s missing using `crypto.randomUUID()`, and injects a structured log line with timestamp, edge POP, URL, referrer, and that ID. The whole thing should be stateless — don’t try to do sessionization *on* the edge node itself.

That log line is what you ship. Don’t ship raw Nginx combined logs; you’ll regret it when you try to parse them later. Emit JSON with exactly what you need: `anon_id`, `session_id` (we derive this as `anon_id + daily salt` to avoid storing long-lived IDs at the edge), `ts_edge`, `url_path`, `referrer_path`, `ua_family`, `pop`, `cache_status`, and `tls_fingerprint`. We push this via Logpush / Log Streaming directly to R2/S3 or to a Vector aggregator in front of Kafka. On a property doing 45k RPS, that’s roughly 2.1TB of clickstream JSON per day uncompressed — about 180GB after gzip. If you’re trying to send that over syslog, you’re going to drop events. For deeper pathing work, you’ll want to join this with your User Journey Analytics from CDN Logs: Mapping the Path to Conversion pipeline so your marketing team can actually see the funnel, not just raw hits.

Here’s the pattern we run on most properties. It’s a Cloudflare Worker snippet, but the logic ports directly to Fastly Rust or Akamai EdgeWorkers. Note we’re not setting a third-party cookie — it’s a first-party, HttpOnly, SameSite=Lax cookie issued from your own domain at the edge:

// Edge clickstream stamping - runs in <5ms
export default {
  async fetch(req, env, ctx) {
    const url = new URL(req.url);
    const cookie = req.headers.get('Cookie') || '';
    let anonId = cookie.match(/__csid=([^;]+)/)?.[1];

    if (!anonId) {
      anonId = crypto.randomUUID();
      // First-party, 13-month expiry, partitioned for CHIPS compliance
    }

    const logEvent = {
      v: 1,
      anon_id: anonId,
      ts_edge: Date.now(),
      pop: req.cf.colo,
      method: req.method,
      path: url.pathname,
      qs: url.searchParams.toString().slice(0, 512),
      referrer_path: req.headers.get('Referer') ? new URL(req.headers.get('Referer')).pathname : null,
      ua: req.headers.get('User-Agent')?.slice(0, 180),
      cache_status: null, // filled after fetch
      asn: req.cf.asn
    };

    const res = await fetch(req);
    logEvent.cache_status = res.headers.get('CF-Cache-Status') || res.headers.get('X-Cache');
    
    // Non-blocking log ship - don't await
    ctx.waitUntil(
      fetch('https://log-ingest.yourdomain.com/clickstream', {
        method: 'POST',
        body: JSON.stringify(logEvent),
        headers: { 'Content-Type': 'application/json' }
      }).catch(() => {})
    );

    const newRes = new Response(res.body, res);
    if (!cookie.includes('__csid')) {
      newRes.headers.append('Set-Cookie', `__csid=${anonId}; Path=/; Max-Age=34164000; HttpOnly; SameSite=Lax; Secure; Domain=.yourdomain.com`);
    }
    return newRes;
  }
}

Advanced Patterns and What Breaks at 10 Billion Requests a Day

Once you’ve got events flowing, the ugly stuff shows up. First is bot contamination. If you don’t filter at ingest, your “most common path” will be `/ -> /robots.txt -> /wp-login.php` and your session lengths will look insane. We’ve seen bot traffic inflate apparent pageviews by 35-60% on unprotected retail sites. Don’t try to do perfect bot detection in the Worker — just flag the signals. Pass along `ja3_hash`, bot score header if you have it (Cloudflare `cf.botManagement.score`, Akamai `Akamai-Bot`), and request rate per `anon_id`. Do the heavy classification downstream in ClickHouse or Druid. We keep a separate pipeline for Bot Traffic Analysis and Classification from CDN Logs: Separating Good from Bad and drop anything with a score <30 or known crawler ASN before we calculate navigation metrics.

Second headache is single-page apps and `fetch` navigation. Your edge sees `/` once, then a bunch of `fetch` calls to `/api/products/123` that don’t map to a pageview. If your frontend uses `history.pushState()`, you won’t have a document request to log. We solve this by having the SPA send a lightweight `navigator.sendBeacon()` to `/_cs/pv` on route change, which the edge then logs as a synthetic pageview with `x-cs-type: spa_transition`. It costs about 400 bytes per transition and gives you 98%+ coverage of soft navs without running a full tag manager. The other gotcha is cache HITs — if you only log at origin, you miss 78-85% of navigations on a well-cached property. Logging at the edge *before* cache is non-negotiable.

Finally, watch your sessionization logic downstream. A fixed 30-minute timeout will artificially split sessions during long reads — we’ve measured a 12% over-count in session fragmentation with that default. Use an adaptive timeout: 30 minutes for active clickstreams, but extend to 4 hours if the user has an active cart or video playing (you can signal that via a `__cs_state` cookie flag). And if you’re attributing revenue, don’t stitch attribution in the same job. Ship the clean clickstream to your warehouse and join conversions later with a separate attribution window. That’s how you keep Conversion Attribution from CDN Edge: First-Party Tracking Without Client-Side Scripts accurate without polluting your navigation graph with last-touch bias.

Frequently Asked Questions