WAF Logging for Forensics: Building an Audit Trail for Security Incidents

A few years back I got paged at 2:13 AM for a suspected data exfiltration on a retail origin we fronted. The WAF had blocked a handful of SQL injection payloads, or so the dashboard said, but when the SOC asked for the full request timeline — headers, rule matches, body hashes, upstream response — we had almost nothing. The default logs showed a timestamp, an IP, and a rule ID. That was it. We couldn't prove what was blocked, what slipped through, or whether the attacker had pivoted. That night taught me you don't build an audit trail during an incident; you build it months before.

Why Your Default WAF Logs Won't Survive a Real Investigation

Most teams turn on WAF logging and assume they're covered. Out of the box, nearly every CDN and WAF vendor gives you a minimal event stream: timestamp, client IP, action (BLOCK/CHALLENGE), and maybe a Ray ID or rule name. That's fine for counting blocks on a dashboard, but it's useless for forensics. When you're doing incident analysis a week later, you need to answer different questions: What was the exact normalized URI? Which parameter triggered the rule? Did the request match one rule or five? Was the body truncated before inspection? Without those details, your security logs are just noise.

I've audited probably two dozen WAF deployments where the logging pipeline was an afterthought. The most common failure I've seen is logging only blocked requests. It sounds efficient — save on storage, reduce volume — until you realize you can't reconstruct attacker behavior. Modern attackers will send hundreds of probes that your WAF deems clean (anomaly scores below threshold) before sending the one payload that gets blocked. If you're only keeping blocks, you've lost the entire reconnaissance phase. You also lose the ability to prove false positives, which matters when legal or compliance asks why a legitimate customer was denied service. Your audit trail has to include allowed traffic, at least in sampled or header-only form, or you're investigating with one eye closed.

A solid forensic log also preserves what the WAF saw, not just what the client sent. That means logging after normalization — URL decoding, Unicode canonicalization, body parsing — alongside the raw request. If you want to understand how your detection actually fired, you need both. This ties directly into how the WAF Rule Engine Architecture: How Inspection Engines Process Requests at Line Speed handles transformations before rule evaluation. Without that context, a rule match on ../etc/passwd looks identical to a match on %2e%2e%2fetc%2fpasswd, and you can't tell if your normalization is working or being abused.

// Bad: What most default WAF logs give you
{
  "timestamp": "2024-11-08T02:13:11Z",
  "client_ip": "203.0.113.45",
  "action": "BLOCK",
  "rule_id": "942100"
}

// Forensic-grade: What you actually need for an audit trail
{
  "timestamp": "2024-11-08T02:13:11.842Z",
  "request_id": "8f3d1a2b-9c4e-4f1a-8b2a-1c9d3e4f5a6b",
  "client_ip": "203.0.113.45",
  "client_asn": 64496,
  "colo": "SJC",
  "method": "POST",
  "raw_uri": "/api/search?q=%2527%20UNION%20SELECT%20*",
  "normalized_uri": "/api/search?q=' UNION SELECT *",
  "headers": {
    "user-agent": "curl/8.5.0",
    "content-type": "application/x-www-form-urlencoded",
    "x-forwarded-for": "203.0.113.45, 198.51.100.10"
  },
  "body_hash_sha256": "e3b0c44298fc1c149afbf4c8996fb924...",
  "body_truncated": false,
  "waf": {
    "matched_rules": [
      {"id": "942100", "phase": 2, "msg": "SQL Injection Attack Detected", "target": "ARGS:q", "maturity": 8},
      {"id": "949110", "phase": 2, "msg": "Inbound Anomaly Score Exceeded", "score": 12}
    ],
    "anomaly_scores": {"sql": 5, "xss": 0, "total": 12},
    "action": "BLOCK",
    "action_taken_by": "managed_ruleset",
    "rule_version": "crs-4.4.0-20241015"
  },
  "origin_response": {"status": 0, "bytes": 0, "latency_ms": 0}
}

Designing a Forensic-Ready Log Schema That Doesn't Fall Apart

We learned the hard way that schema discipline matters more than storage. Back when we ran a multi-CDN setup for a finance client, each edge provider sent logs in a slightly different JSON shape. One used ruleId, another rule_id, a third nested everything under waf. Our SIEM normalized on ingest, and that normalization silently dropped fields when a vendor shipped an update. We didn't notice until an incident where half our matched rule IDs were null in Splunk. Now I insist on a canonical schema at the collector layer, before anything hits long-term storage. You want a single, versioned definition for your WAF logging events and you want to enforce it with validation, not hope.

The schema itself should balance completeness with queryability. For forensics, I treat five groups as non-negotiable: request identity (request ID, edge colo, timestamp with millisecond precision), client context (IP, ASN, JA3/JA4 fingerprint, TLS version), request data (method, host, raw and normalized URI, headers, body hash), WAF verdict (all matched rules, anomaly scores, phase, action and which component took it), and origin outcome (status, latency, bytes). Don't store full bodies in hot storage — hash them and archive the compressed body to object storage linked by request ID. That keeps your SIEM fast while still letting you pull the actual payload during incident analysis. If you're following modern HTTP semantics, make sure header names are lowercased and preserve the order you received them; the HTTP Working Group Specifications are clear on header field case-insensitivity, but attackers love to use weird casing to dodge naive parsers, and you'll want to see that.

One thing that saves hours during a post-mortem is deterministic request correlation. Every log line — WAF, CDN cache, origin, and application — should share the same request_id. Generate it at the edge on first contact and propagate it as x-request-id to origin. Then your audit trail isn't three separate puzzles; it's one timeline. We've also started logging the WAF Managed Ruleset Comparison: OWASP CRS, Commercial, and Provider-Specific Rules version and paranoia level with every event. Sounds trivial, but when you're comparing why a payload was blocked on Tuesday and allowed on Thursday, that version string is the first thing you'll check. Pair that with structured logging guidance from web.dev Articles on preserving client hints and request metadata, and you've got logs a junior analyst can actually trace without guessing.

# Vector / Logpush pipeline: enforce schema + hash body + centralize
[transforms.waf_enrich]
type = "remap"
inputs = ["edge_waf_raw"]
source = '''
  .request_id = get_env_var("EDGE_REQUEST_ID") ?? uuid_v4()
  .timestamp = to_timestamp!(.timestamp, unit: "milliseconds")
  
  # Canonicalize header keys and preserve order for forensics
  .headers_normalized = map_keys(.headers) -> |key| { downcase(key) }
  
  # Hash body for audit trail, don't ship raw PII to SIEM
  if exists(.body) {
    .body_hash_sha256 = sha2(.body, variant: "SHA-256")
    .body_truncated = length(.body) > 131072
    del(.body)
  }
  
  # Validate against versioned schema - drop or quarantine non-conforming
  if !is_valid_schema(., version: "waf_forensic_v3") {
    .quarantine_reason = "schema_validation_failed"
    . = tag(., "quarantine")
  }
  
  # Add WAF context for audit trail
  .waf.rule_version = "crs-4.4.0"
  .waf.collector_version = "waf_forensic_v3"
'''

[sinks.siem_hot]
type = "elasticsearch"
inputs = ["waf_enrich"]
index = "waf-forensic-%Y-%m-%d"

[sinks.archive_raw]
type = "aws_s3"
inputs = ["waf_enrich"]
bucket = "waf-audit-archive"
compression = "gzip"
encoding.codec = "json"

Storage, Sampling, and Retention: Making Forensics Affordable

Here's where most engineering managers push back: "We can't store every request with full headers for 365 days." And they're right, you can't — at least not in hot storage. A mid-size property doing 20k requests per second will generate around 1.7 billion events a day. At 1.2KB per forensic log, that's roughly 2TB per day before compression. Keep that in Elasticsearch for a year and you'll fund your vendor's next yacht. The trick isn't to log less; it's to tier your storage and sample intelligently.

What we've settled on after a lot of trial and error is a three-tier approach. All WAF events — blocks, challenges, and anomalies — go to hot storage for 30 days with full detail. Allowed requests get sampled: 100% for the first 1k requests per minute per edge, then 10% random sampling plus 100% of any request that scores even 1 point on an anomaly scale. That catches slow-and-low scans that never trip a block. Full bodies and raw headers go to S3/GCS with GZIP + ZSTD for 13 months, keyed by request ID and retrievable in under a minute. For our team, that cut SIEM ingest by 68% without losing forensic coverage, because we could still rehydrate any incident window from the archive.

You'll also want to define retention based on your actual threat model, not just compliance checkboxes. PCI DSS might say one year, but if you're running a WAF Custom Rule Authoring: Writing Effective Detection Rules Without False Positives workflow where you're frequently tuning and rolling back rules, you need at least 90 days of hot data to compare false positive rates before and after a change. And don't forget to log rule changes themselves — who published what, when, and the diff. Your WAF logs show what the engine did; your rule audit log shows why it was configured that way. During our last red team exercise, the most valuable artifact wasn't the block log, it was the Git history of the custom rule that almost let a bypass through.

From Schema to Pipeline: Shipping Logs Without Tanking Performance

Alright, you've got a solid schema on paper — now you’ve got to actually get it off the edge and into somewhere useful without adding 40ms of latency to every request. The biggest mistake I see teams make is trying to log synchronously. Don't do it. Your WAF should be writing to a local ring buffer or shared memory queue and letting an async shipper handle delivery. If your log pipeline blocks the request path, you've just built yourself a very expensive DoS vector.

We typically run Vector or Fluent Bit as a sidecar on every edge PoP, tailing the WAF's JSON log socket. It handles backpressure, retries, and — crucially — PII scrubbing before anything leaves the node. You don't want raw `Authorization` headers or credit card numbers sitting in your SIEM for 90 days. Scrub at the edge, not in the pipeline. A good setup will buffer at least 500MB or ~2 million events locally, so you can survive a 15-20 minute outage of your central collector without dropping forensic data. Anything less and you’ll lose the exact window you need during a real incident.

Here’s the skeleton we use for a forensic-grade log event. It’s verbose, but that verbosity is what lets you reconstruct an attack three weeks later without guessing. Notice we’re capturing not just the rule that fired, but the full transaction context — that’s non-negotiable.

{
  "event_id": "01H9K6QZ..._iad_42",
  "timestamp": "2026-05-13T14:22:31.142Z",
  "edge_node": "iad-42",
  "request": {
    "method": "POST",
    "uri": "/api/v1/login",
    "headers_truncated": false,
    "ja3_hash": "e7d705a3286e19ea43f81d...",
    "asn": 15169,
    "country": "US",
    "bot_score": 23
  },
  "waf": {
    "action": "BLOCK",
    "rule_id": "941330",
    "rule_family": "CRS_sqli",
    "anomaly_score": 8,
    "matched_data": "SELECT @@version",
    "processing_ms": 0.7
  },
  "correlation": {
    "ray_id": "8a3f1c9e2b",
    "rate_limit_bucket": "login:192.0.2.14",
    "session_id": "sess_abc123"
  }
}

Advanced Correlation and Troubleshooting When Your Timeline Has Gaps

Even with perfect logging, investigations fall apart when you can't correlate. Your WAF log says it blocked an IP at 14:22:31.142Z, but your origin log shows a 200 OK at 14:22:31.189Z for the same URI. Which one’s lying? Usually neither — you’ve got clock skew. We’ve seen 800ms drift between edge and origin kill an entire forensic timeline. Sync everything to NTP with PTP if you can, and always log `ray_id` or `x-request-id` at both layers. Without a single correlation ID, you’re just guessing which blocked request maps to which application error.

The other killer is missing context. If you’re sampling aggressively to save costs, you *must* sample intelligently. Don’t randomly drop 80% of 200s. Keep 100% of all blocks, challenges, and anything with an anomaly score >5, plus 1% of clean traffic for baseline. That keeps your hot storage down from ~3.2 TB/day at 10k RPS to around 180-250 GB/day while preserving every event you’ll actually need. And when you’re trying to link WAF events to abusive behavior, you’ll want that bot score and rate-limit state right in the log — don’t make analysts join three systems later. Integrating those signals up front is why we strongly recommend reading WAF Bot Score Integration: Using Bot Detection Signals in Firewall Rules and WAF Rate Limiting Integration: Combining Application Rules with Traffic Controls before you finalize your schema.

Finally, if your logs are noisy, your forensics are useless. I’ve audited setups where 94% of blocks were false positives on a single CRS rule — analysts just started ignoring them. If you can’t trust your block log, you don’t have an audit trail. You need a tuning loop. After any major incident or weekly if you’re under constant probe, pull your top 10 rule IDs by block volume, sample 100 events each, and validate. Our process for that is detailed in WAF False Positive Tuning: A Systematic Methodology for Reducing Alert Noise — it’ll cut your noise by 60-70% in the first two passes and makes the remaining logs actually investigable.

Frequently Asked Questions