Certificate Transparency Log Monitoring: Automated Detection of Rogue Certificates

At 2:37 AM on a Tuesday in late 2021, my on-call phone buzzed with an alert I'd never seen before: a certificate for `api.payments.ourcustomer.example` had appeared in a Certificate Transparency log, issued by a CA we’d never used. No ticket, no change request, no approval. That cert was valid, publicly trusted, and would have passed any browser check. If we hadn’t been tailing CT logs in near-real-time, that rogue certificate could have sat there for weeks, waiting to be used for interception. That night changed how our team thinks about Certificate Lifecycle Automation: From Issuance to Renewal at CDN Scale forever.

Why Traditional PKI Validation Misses Rogue Certificates Entirely

For years, most ops teams treated PKI security as a validation problem — check the chain, check the expiry, check the revocation via CRL or OCSP, and you’re good. I did the same for most of my early CDN career. The problem is that a rogue certificate isn’t invalid in the cryptographic sense. It’s been signed by a legitimate, trusted CA, its chain validates perfectly, and it’ll sail through the checks described in the HTTP Working Group Specifications (https://httpwg.org/specs/). Your edge nodes will happily serve it, and OCSP Stapling for CDN: Eliminating Certificate Validation Delays will even make it faster by stapling a nice “good” status right to the handshake.

CT log monitoring flips that model on its head. Instead of asking “is this certificate valid?” we ask “should this certificate exist at all?” When you’re operating a CDN that fronts 400+ customer domains, you can’t rely on CAs to prevent mis-issuance. I’ve seen Domain Validation bypassed through hijacked DNS TXT records, stolen ACME account keys, and once, through a reseller’s buggy automation that issued a wildcard for the wrong base domain. All of those resulted in perfectly valid certificates that shouldn’t have existed. The only place they all showed up was in the public CT logs, because since Chrome’s enforcement in 2018, any publicly trusted cert must be logged or browsers will reject it.

That’s the core insight: CT is our only global, append-only audit trail. If you aren’t watching it, you’re blind to unauthorized issuance. A simple poll of crt.sh like the one below was our first detector before we built streaming. It’s crude, but it catches what OCSP and CRLs never will.

#!/usr/bin/env python3
import requests
import time
from datetime import datetime, timedelta

# Poll crt.sh for certificates matching our monitored domains
# This is the polling approach we used before moving to CertStream
MONITORED_SUFFIXES = ["example.com", "example.net"]
CRT_SH_URL = "https://crt.sh/?q=%.{domain}&output=json"

def check_for_rogue_certs(seen_serials):
    for domain in MONITORED_SUFFIXES:
        try:
            resp = requests.get(CRT_SH_URL % domain, timeout=10)
            resp.raise_for_status()
            for entry in resp.json():
                serial = entry['serial_number']
                if serial not in seen_serials:
                    # Alert if issuer is not in our allowlist
                    issuer = entry['issuer_name']
                    if "Let's Encrypt" not in issuer and "DigiCert" not in issuer:
                        print(f"[{datetime.utcnow()}] ROGUE? {entry['common_name']} "
                              f"issued by {issuer} serial:{serial}")
                    seen_serials.add(serial)
        except Exception as e:
            print(f"Poll failed for {domain}: {e}")
    return seen_serials

# Run every 10 minutes - note the inherent lag vs streaming
if __name__ == "__main__":
    seen = set()
    while True:
        seen = check_for_rogue_certs(seen)
        time.sleep(600)

Building a Real-Time CT Log Ingestion Pipeline That Doesn't Drop Entries

Polling crt.sh works for a proof-of-concept, but it’s not monitoring. The lag is 15-30 minutes at best, you’ll get rate-limited, and you’ll miss precertificates — those pre-issued entries CAs log before the final certificate, which are often the earliest signal of mis-issuance. When we scaled to watching 50k+ hostnames across our CDN customers, polling fell over completely. We needed a streaming pipeline that could keep up with every major log: Google Xenon, Cloudflare Nimbus, DigiCert Nessie, and Let’s Encrypt Oak.

Our current stack is pretty boring, and that’s intentional. We consume the CertStream WebSocket feed (which aggregates multiple logs) and also tail a couple of logs directly via the `ct` Go client for redundancy, because CertStream has had outages. Everything lands in Kafka, then a stateless worker does three things: 1) normalize the SANs and commonName, 2) match against our inventory of authorized domains and wildcard scopes, and 3) verify the issuer against an allowlist per customer. If you’re choosing between Wildcard vs SAN Certificates for CDN: Making the Right Choice, this is where that decision matters — a rogue `*.example.com` is vastly more dangerous than a single SAN entry, and your matching logic has to understand that scope. We also learned to parse the precertificate poison extension correctly; early on we flagged every precert as a duplicate and alert fatigue buried the real alerts.

One mistake I see teams make is trusting log order. CT logs are Merkle Trees, not strict queues. Entries can appear slightly out of order and `get-entries` can serve stale STHs (Signed Tree Heads) for a few seconds. If your code assumes strictly increasing indices, you’ll skip entries during a log shard rollover. The web.dev Articles (https://web.dev/articles/) guidance on CT log integration covers this, but the fix in practice is simple: always verify the `tree_size` and re-fetch with back-off. The worker below shows the core matching logic we run on every cert we see — about 8-12 million certs per day now.

import re
import json
from fnmatch import fnmatch

# Example allowlist: customer_id -> { allowed_issuers, allowed_wildcards }
ALLOWLIST = {
    "customer_814": {
        "issuers": ["Let's Encrypt", "Google Trust Services"],
        "domains": ["example.com", "*.example.com", "api.example.com"]
    }
}

def is_authorized(cert_sans, issuer_org, customer_id):
    policy = ALLOWLIST.get(customer_id)
    if not policy:
        return False  # No policy = treat as rogue until triaged

    # 1. Check issuer is expected
    if not any(allowed in issuer_org for allowed in policy["issuers"]):
        return False

    # 2. Check every SAN is within allowed scope
    for san in cert_sans:
        san = san.lower()
        if not any(fnmatch(san, pat.lower()) for pat in policy["domains"]):
            # e.g., san=admin.example.com matches *.example.com -> ok
            # san=evil.example.com.attacker.com -> fails
            return False
    return True

# Called for each message from CertStream/Kafka
def handle_ct_entry(entry):
    sans = entry['leaf_cert']['all_domains']
    issuer = entry['leaf_cert']['issuer']['O']
    # Reverse lookup customer by domain suffix
    customer_id = lookup_customer(sans[0]) 
    if not is_authorized(sans, issuer, customer_id):
        trigger_alert(entry, reason="Issuer or SAN not in allowlist")

def lookup_customer(domain):
    # Simplified inventory lookup
    return "customer_814" if domain.endswith("example.com") else None

From Noise to Signal: Tuning Detection Without Missing Real Threats

When we first turned on streaming alerts, we got hammered. Over 600 alerts in the first 48 hours, and exactly one was a real rogue — a pentest team’s authorized issuance we hadn’t been told about. The rest were noise: Let’s Encrypt duplicates due to retries, Google Trust Services re-issuing with different precert serials, and a long tail of dev certificates for `*.staging.example.com` that the customer forgot to add to their allowlist. If you alert on everything, your team will mute the channel, and you’ll miss the one that matters at 2 AM.

We fixed it by adding two layers of filtering before the human ever sees an alert. First, an automatic reconciliation job that compares every new CT entry against our own ACME and certificate management database. If we issued it through our automation, it’s auto-closed in under 2 seconds with a log entry. No page. Second, we score what’s left: wildcard issuance is critical, EV/OV mismatch is high, issuance from a new CA for that customer is high, and a precert that never turns into a final cert is low but still logged. That cut our false positive rate by over 90% and brought mean time to triage down to minutes. We also enforce CAA records aggressively now; they don’t prevent all mis-issuance, but they shrink the set of CAs we need to watch.

The numbers tell the story better than I can. After six months of tuning, this is where our pipeline settled versus the naive polling approach we started with. We process logs from 6 major operators and keep a 90-day hot store in ClickHouse for forensic searches — something you’ll want when a customer asks “has anyone ever issued for this subdomain before?”

Putting It Into Practice: Automating Rogue Issuance Alerts at the Edge

So you've got a CT firehose running — now what? Raw ingestion is useless without automated triage that actually wakes someone up. At our edge, we don't just log every cert for *.example.com; we run a matching engine that checks every new precertificate and leaf against a normalized inventory of owned domains, PSKs, and SAN variants, including IDN punycode and wildcard expansions. If you've got 400+ domains across 12 TLDs like we do, you can't rely on exact string matches. We use a combination of exact match, Levenshtein distance <=2 for typo-squatting, and a 4-hour lookback for newly registered lookalikes via WHOIS diff.

Once a match fires, speed matters. A rogue cert is typically weaponized within 15-45 minutes of issuance — we've seen phishing kits go live in under 9 minutes. Your pipeline needs to push from CT log entry to PagerDuty in under 60 seconds. We do this with a lightweight Go service sitting behind certstream that evaluates, enriches with WHOIS and CAA record checks, and posts to a webhook. Anything with high confidence auto-creates a ticket, triggers an ACME revocation check, and temporarily blocks that cert's SPKI hash at the edge via our Beyond Certificate Pinning: Modern Alternatives for CDN Trust Verification policy engine. Low-confidence matches get batched to Slack for human review — cuts our alert fatigue by about 83% compared to alerting on everything.

// ct-monitor.go - simplified CT log watcher with SPKI blocking
package main

import (
  "log"
  "strings"
  "github.com/CaliDog/certstream-go"
)

var ownedDomains = map[string]bool{"example.com": true, "examplecdn.net": true}
var blockedSPKI = make(map[string]bool)

func handleCert(cert *certstream.CertificateEntry) {
  for _, domain := range cert.Data.LeafCert.AllDomains {
    base := strings.ToLower(strings.TrimPrefix(domain, "*."))
    if ownedDomains[base] || isLookalike(base) {
      spkiHash := cert.Data.LeafCert.SPKISHA256()
      log.Printf("[ROGUE] %s issued by %s - SPKI:%s", domain, cert.Data.LeafCert.Issuer.O, spkiHash[:16])

      // Auto-block at edge + alert
      if confidenceScore(domain) > 0.85 {
        blockAtEdge(spkiHash) // push to edge KV in <200ms globally
        triggerPagerDuty(domain, cert)
      } else {
        sendToSlack(domain, cert)
      }
      
      // Enrich with handshake data for forensics
      enrichTLSFingerprint(domain)
    }
  }
}

func main() {
  stream, _ := certstream.CertStreamEventStream(false)
  for ev := range stream {
    if ev.MessageType == "certificate_update" {
      handleCert(ev.Data)
    }
  }
}

Troubleshooting at Scale: When Logs Lag, Duplicates Flood, and Signals Get Lost

Here's where most teams get burned: CT logs aren't perfectly real-time. We've measured Argon2024 and Xenon shards lagging 35-90 seconds behind wall-clock during peak issuance windows (typically 14:00-16:00 UTC, when Let's Encrypt pushes ~2.8M certs/hour). If your monitor only tails one log, you'll miss 22-30% of precertificates that hit Google's logs first. Don't tail ct.googleapis.com/logs/argon2024 in isolation — fan out to at least 4-5 active logs and deduplicate by tbsCertificate hash. We also add a 5-minute reconciliation job that polls crt.sh for any certs we didn't see streaming; that catches another 1.5% that slipped through websocket drops.

The other killer is noise from precertificates vs. final certs. Every issuance emits two entries with the same serial, and naive pipelines will double-alert. Filter on x509v3 Extended Key Usage and the poison extension (OID 1.3.6.1.4.1.11129.2.4.3) to only process precerts once, then suppress the final leaf for 24 hours. And if you're seeing false positives from CDN-managed issuance — like your own edge provider re-issuing certs on rotation — maintain an allowlist of your authorized CA account IDs and CAA-authorized issuers. We learned this the hard way after our own auto-renewal triggered 47 alerts in one night.

Finally, don't ignore the transport layer once you've detected something. A rogue cert is often paired with a cloned origin behind cheap hosting that doesn't tune TLS properly. We've found correlating CT alerts with anomalous TLS Fingerprinting Defense: Preventing Client Identification Through Handshake Analysis JA3 hashes and odd TLS Record Size Optimization: Balancing Latency and Throughput at the Edge chunking (like 4KB records on a phishing nginx) gives us a 4x higher true-positive rate. If the handshake looks nothing like your real edge — say, RSA-2048 where you only deploy ECDSA P-256 — that's your confirmation to escalate from "suspicious" to "revoke now."

Frequently Asked Questions