Bot Traffic Analysis and Classification from CDN Logs: Separating Good from Bad

Last November I was staring at a dashboard at 2 a.m. before Black Friday and watched what looked like a beautiful 40% traffic spike roll in. Our concurrent users were up, edge requests were through the roof, we should’ve been celebrating. Instead our conversion rate flatlined, origin latency doubled, and cache hit ratio dropped twelve points in an hour. When we finally tore into the raw CDN edge logs, almost sixty percent of that “growth” wasn’t human at all — it was bots we’d been counting as customers for months.

Your Edge Logs Know More Than Your Analytics Dashboard

I’ve learned the hard way that your JavaScript analytics will always undercount bots. Good bots don’t run your JS, bad bots actively avoid it, and the stuff that hammers your checkout API never touches a browser. Your CDN logs are the only place you see everything. Every request, every TLS handshake, every cache status, every blocked challenge — it’s all there if you’re collecting it right. The trouble is most teams are still shipping logs once an hour to an S3 bucket and trying to do forensics the next day. By then the scraper has already taken your entire catalog.

What you actually need in each log line is richer than the classic combined log format. User-Agent alone is useless now — I’ve seen credential stuffing botnets that rotate through 10,000 perfect Chrome UAs per minute. You need client IP and ASN, JA3 or JA4 fingerprint, TLS version, cache status (HIT/MISS/PASS), request rate per IP over a sliding window, and the response size. When we rebuilt our pipeline we moved to a proper CDN Log Aggregation Architecture: Collecting Billions of Events Without Dropping Data so we wouldn’t lose events during peaks, and that change alone showed us 18% more automated traffic we’d been dropping on the floor.

A single raw line tells you more than you think once you know what to look for. Here’s a simplified JSON event from our edge, with the fields I actually use for classification:

{
  "timestamp": "2024-11-14T02:17:43.112Z",
  "edge_pop": "ord-07",
  "client_ip": "192.0.2.88",
  "asn": 396982,
  "ja3_hash": "e7d705a3286e19ea42f587b344ee6865",
  "method": "GET",
  "uri": "/api/products/88473/reviews?page=112",
  "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
  "referer": "-",
  "cache_status": "MISS",
  "status": 200,
  "bytes_out": 18432,
  "req_per_min_ip": 247,
  "tls_version": "TLSv1.3"
}

How I Build a Bot Classification Pipeline That Actually Works

I don’t trust any single signal. Ten years ago you could block by User-Agent regex and call it a day. Now you can’t. I use a layered approach that starts simple and gets stricter. First pass is the allowlist — verified good bots that prove ownership. Googlebot, Bingbot, Applebot — they all publish IP ranges and support reverse DNS verification. If the IP doesn’t validate, it’s not Google, no matter what the UA says. Second pass is behavior. Humans don’t request 247 paginated API pages per minute with no referer and no cookies, and they don’t consistently get MISS because they’re crawling for fresh data. That’s why capturing req_per_min_ip at the edge is so critical.

The second piece I’ve added to every recent setup is fingerprinting and rate accounting before it even hits your origin. JA3/JA4 hashes paired with HTTP/2 fingerprinting catch headless browsers that spoof UAs perfectly but use the same TLS library. And you’ve got to do this in real time. Batch jobs are great for reporting, but blocking has to happen within seconds. We switched to a CDN Real-Time Log Streaming Architecture: From Edge Events to Analytics in Seconds using Kafka at the POPs and ClickHouse in the middle, which let us run sliding window counts and push decisions back to the edge via API in under five seconds. Before that, our “real-time” blocklist was fifteen minutes stale and the bad bots were long gone.

Here’s a ClickHouse query I use as a starting classifier for triage. It’s not production ruleset, but it buckets traffic so you can see what you’re dealing with:

SELECT
  client_ip,
  asn,
  ja3_hash,
  count(*) AS reqs,
  avg(req_per_min_ip) AS avg_rpm,
  uniqExact(uri) AS unique_urls,
  countIf(cache_status = 'MISS') / count(*) AS miss_ratio
FROM cdn_edge_logs
WHERE timestamp >= now() - INTERVAL 15 MINUTE
GROUP BY client_ip, asn, ja3_hash
HAVING reqs > 500 AND miss_ratio > 0.85
ORDER BY reqs DESC
LIMIT 100;

Separating the Good, the Bad, and the Expensive Grey Area

Not all bots are bad. That’s where teams mess up and nuke their SEO. Good bots identify themselves, respect robots.txt, throttle themselves, and come from verifiable infrastructure — think Googlebot, Bingbot, or Slack’s unfurler. You want them, you just don’t want them counted in your business metrics or allowed to hammer your expensive origin API uncached. Bad bots are the opposite: they lie about who they are, rotate IPs across cheap hosting ASNs, ignore robots.txt defined in the HTTP Working Group Specifications, and generate the high miss ratios and odd header patterns you’ll see in the query above. The web.dev Articles on crawl budget and bot handling are a decent primer if you haven’t tuned robots.txt and Cache-Control for crawlers before.

The grey area is what’ll eat your budget. I’m talking about “good-ish” bots — price monitors your commercial team approved, affiliate scrapers, SEO tools like Ahrefs and Semrush, and AI training crawlers. They’re not malicious, but they can be just as heavy as an attack if you let them run wild. I usually put them in a throttled bucket: allow, but serve from cache, rate limit at the edge, and never let them hit dynamic APIs. And don’t forget to think about sampling costs. Logging every request 1:1 is ideal for security, but at 50k RPS it’ll bankrupt your observability bill. We use tuned CDN Log Sampling Strategies: Reducing Storage Costs Without Losing Signal — full fidelity for 4xx/5xx and MISS traffic, 10:1 sampling for cache HITs from known good ASNs — so we keep signal without drowning in data.

When I run a bot audit for a client, this is the breakdown I start with. Your numbers will vary, but the pattern is remarkably consistent:

From Theory to Edge: Implementing Real-Time Bot Scoring

Reading logs after the fact is useful for reporting, but it won't save your origin when a scraper swarm decides to hit you. The real win is scoring every request at the edge, within 5-15ms, and making an immediate routing decision. My production setup is pretty boring, which is exactly why it works: CDN logs streamed via Logpush/Syslog to a Vector aggregator, parsed into ClickHouse, and a lightweight scoring service running as a Cloudflare Worker / Fastly Compute function that reads the same ruleset. We’re processing about 1.8 billion log lines a day on this — you don’t need Spark for it, you need a tight schema and aggressive sampling on the obvious human traffic.

Every request gets a composite bot score from 0-100 before it ever hits origin. I don’t rely on a single signal. I weight four buckets: fingerprint consistency (does the JA3 / HTTP/2 fingerprint match the declared User-Agent?), behavioral velocity (requests per second, page depth, asset vs. HTML ratio), network reputation (ASN, datacenter vs. residential, IP velocity across our entire customer fleet), and verification (reverse DNS + forward-confirmed lookup for the good bots). A verified Googlebot passing rDNS might score a 5, while a headless Chrome hitting 120 product pages per minute from a Hetzner ASN with no image/CSS requests scores a 94. That separation is critical because your Clickstream Analysis at CDN Edge: Understanding User Navigation Patterns is completely worthless if 40% of those “user journeys” are just bots inventorying your catalog.

The trick is to make the score actionable without being destructive. We don’t hard-block at 70. We tier it: 0-30 = pass, 31-69 = soft challenge (managed JS challenge that real browsers solve in <40ms), 70-89 = hard CAPTCHA + rate limit to 10 req/min, 90+ = block at edge with a 444 and log it. This cut our originShield egress by 38% in the first two weeks and dropped our false positive rate from 2.1% to 0.3% because we stopped treating verification as a binary block/allow.

# simplified edge scoring logic (Python-style pseudocode for Worker/Compute)
def score_request(req, ip_meta, verified_bots):
    score = 0
    
    # 1. Verification check - don't penalize good bots
    if req.ip in verified_bots and verify_rdns(req.ip, req.ua):
        return 5  # verified Googlebot, Bingbot, etc.

    # 2. Fingerprint mismatch is a huge signal
    if tls_fingerprint_mismatch(req.ja3, req.ua):
        score += 35
    if req.asset_ratio < 0.15: # HTML only, no CSS/JS/images
        score += 25

    # 3. Behavioral velocity
    rps = get_rps(req.ip, window="60s")
    if rps > 40: score += 30
    elif rps > 10: score += 15
    
    # 4. Network reputation
    if ip_meta.asn_type == "hosting" and rps > 5:
        score += 20
    if ip_meta.datacenter_score > 0.85:
        score += 15

    return min(score, 100)

When Your Rules Stop Working: Evasion, False Positives, and Grey Area Hell

If you built your bot detection in 2021 and haven't touched it, you're likely missing about half the bad traffic now. The cheap curl-based scrapers are gone. What I’m seeing daily is residential proxy rotation (think 50k+ IPs from Bright Data or Oxylabs), real Chrome with Puppeteer Stealth, and bots that actively mimic human pacing — they’ll request the HTML, wait 1.2-3.4 seconds, fetch the CSS/JS, scroll, then click. They even solve basic JS challenges. Your old rule of “blocks if no cookie support” doesn’t catch them. We catch them by correlating User Journey Analytics from CDN Logs: Mapping the Path to Conversion with entropy analysis. Humans are messy. They bounce, they backtrack, they add to cart and abandon. Bots are too perfect. When I see 800 sessions with identical 7-page depth, 0 variance in time-on-page (always 2.1s ±0.1s), and 100% conversion to the JSON API endpoint — that’s not a user segment, that’s automation.

The other nightmare is false positives that cost you money. We once tar-pitted an entire corporate office because their Zscaler egress IP was making 600 req/min with a single JA3 — it looked like a botnet, but it was 300 actual employees behind one NAT. That’s why you never auto-block on IP velocity alone. You have to combine ASN type (enterprise vs. hosting), TLS fingerprint diversity (300 users = 40+ JA3s, a botnet = 1-2), and session cookie persistence. Now we fingerprint at the /32 and /24 level and require at least 3 signals before we escalate to a block. It’s also why your attribution has to be bot-aware — if you’re feeding bot traffic into your marketing model, you’re making budget decisions on garbage. A clean pipeline feeds directly into Conversion Attribution from CDN Edge: First-Party Tracking Without Client-Side Scripts so you’re not crediting a scraper for “viewing” your pricing page.

Finally, watch your grey area bots — the ones that claim to be “good” but hammer you. Think AI crawlers, SEO tools, and price comparison engines. They obey robots.txt (sometimes), they identify themselves, but they’ll still pull 4TB of images a week and never convert. Don’t treat them as malicious, treat them as expensive. We handle this with a separate budget: verified grey bots get a 2 req/sec token bucket, served stale-from-cache, and deprioritized to our lowest-tier cache nodes. They get their data, we don’t pay for origin fetches, and real users keep their 35ms TTFB.

Frequently Asked Questions