CDN Postmortem Analysis: Learning from Incidents to Prevent Recurrence

At 02:17 UTC on a Tuesday, our edge in Ashburn started returning 502s for 18% of image requests, and within twelve minutes that failure had rippled to five other PoPs. The on-call engineer purged the cache, traffic recovered, and we all told ourselves we'd fixed it. Three weeks later the same pattern knocked out video delivery during a major live event, because our first postmortem had stopped at "stale cache entry" and never asked why that entry was allowed to poison the entire tier in the first place. If you've run a CDN long enough, you've lived through that exact loop where a quick fix hides a deeper design flaw.

Why "Human Error" Isn't a Root Cause in CDN Postmortems

I've read maybe two hundred CDN postmortems over the past ten years, and the ones that don't prevent recurrence almost always end with the same line: "Root cause: operator pushed incorrect config." That's not a root cause, that's a starting point. When we stop there, we miss the system conditions that made that mistake both possible and impactful. Real SRE Practices for CDN: Applying Site Reliability Engineering to Content Delivery treat human action as a symptom of tooling and process gaps, not the final answer. In our Ashburn incident, the engineer *did* push a header rule that stripped Surrogate-Key, but the more important questions were why our validation pipeline allowed a rule that violates cache key semantics, and why a single PoP's misconfiguration could propagate globally without a circuit breaker.

I learned this the hard way back in 2018 when I approved a VCL change that accidentally set TTL to 0 for all .m3u8 manifests. We blamed the review process for about a week until we realized our staging environment didn't actually replay real HLS traffic - it only tested synthetic health checks. We'd built a system where you couldn't safely test what you were about to ship, so of course someone eventually shipped a bad change. The fix wasn't "be more careful," it was building a shadow traffic replay that validates cache behavior against production request samples and checks compliance with the HTTP Working Group Specifications for cache-control handling. That shift from blaming people to fixing the system is what separates a useful postmortem analysis from a document that just collects dust.

Effective incident learning means pushing past the first plausible explanation with structured techniques. I use a constrained "5 Whys" for CDN incidents that forces each answer to point to a controllable system factor, not a person. You also need to differentiate between the trigger, the contributing factors, and the amplifying mechanisms. For CDN incidents, amplifying mechanisms are often things like inconsistent negative caching, origin shielding failures, or a purge queue that silently backs up. If your postmortem doesn't name those explicitly, your action items won't address them.

# CDN Postmortem - Root Cause Template (anti-pattern vs. correct)
incident: "ASH-2024-11-02 Cache Poisoning"

# BAD - stops at human action
root_cause_bad: "Engineer deployed invalid surrogate-key rule"

# GOOD - forces systemic analysis
five_whys:
  1_why: "Edge returned stale poisoned objects for 18% of requests"
     answer: "Surrogate-Key header stripped, so purge could not invalidate"
  2_why: "Why was stripping allowed?"
     answer: "Config linter only checked syntax, not cache semantics"
  3_why: "Why didn't validation catch it?"
     answer: "Staging has no production traffic replay for header logic"
  4_why: "Why did one PoP poison the others?"
     answer: "Origin shield fetched poisoned object and replicated via tiered cache"
  5_why: "Why no containment?"
     answer: "No checksum validation on cached objects before tier replication"

systemic_fixes:
  - "Add semantic lint: block rules that modify Surrogate-Key/Cache-Tag"
  - "Require shadow replay validation before global push"
  - "Enable object integrity hash check in tiered cache fill"

Reconstructing the Timeline: What Good Incident Evidence Looks Like

Most teams build their timeline from Slack messages and human memory, which means you're reconstructing what people *thought* happened, not what the system actually did. For CDN incidents, the truth lives in at least four data sources that rarely agree at first glance: edge access logs, control plane deployment logs, origin health signals, and client-side RUM. On that Ashburn incident, our chat timeline said "purge completed at 02:29," but the purge queue metrics showed the API returned 202 Accepted while the queue depth was actually growing from 4k to 47k jobs. The purge hadn't completed, it had just been accepted. If we'd trusted chat over metrics, we'd have concluded purges work fine and missed the core failure mode entirely.

You've got to correlate these signals with synchronized clocks and request tracing. I now insist that every postmortem includes a unified timeline with millisecond-precision timestamps from NTP-synced logs, plus the cache status field (HIT, MISS, REVALIDATED, BYPASS) for representative requests. The web.dev Articles on caching best practices are useful here for aligning what your CDN *should* do with cache-control and stale-while-revalidate against what it *did* do. And don't forget to tie your timeline to error budget impact. When we link incident minutes to our CDN Error Budget Management: Balancing Reliability with Feature Velocity burn rate, it changes the conversation from "it was only 12 minutes" to "we burned 34% of our quarterly budget in one event," which gets proper prioritization for fixes.

A practical trick that's saved me hours is to script the timeline extraction before the postmortem meeting even starts. Don't make senior engineers manually grep logs for three hours - automate the evidence collection so the meeting can focus on interpretation. That reduction in manual work is part of broader CDN Toil Reduction Through Automation: Eliminating Repetitive Operations Work, but for postmortems it has a direct quality benefit: you get consistent data every time, and you avoid the hindsight bias that creeps in when people reconstruct events days later. Your future self will thank you when you need to compare similar CDN incidents from six months apart.

-- Reconstruct CDN incident timeline from structured logs
-- Correlate edge errors, config pushes, and purge queue depth

WITH edge_errors AS (
  SELECT 
    date_trunc('minute', timestamp) AS minute,
    pop,
    COUNT(*) FILTER (WHERE status >= 500) AS errors_5xx,
    COUNT(*) FILTER (WHERE cache_status = 'HIT' AND age > 3600) AS stale_hits
  FROM cdn_access_logs
  WHERE timestamp BETWEEN '2024-11-02 02:10:00' AND '2024-11-02 03:00:00'
  GROUP BY 1, 2
),
control_plane AS (
  SELECT timestamp, actor, action, config_version
  FROM cdn_config_audit
  WHERE action IN ('push', 'purge_request')
)
SELECT e.minute, e.pop, e.errors_5xx, e.stale_hits, c.action, c.config_version
FROM edge_errors e
LEFT JOIN control_plane c ON c.timestamp BETWEEN e.minute AND e.minute + INTERVAL '1 minute'
WHERE e.errors_5xx > 100
ORDER BY e.minute;

Turning Action Items Into Systemic Fixes That Prevent Recurrence

Here's where most postmortem analysis falls apart: we create action items like "add monitoring" or "update runbook" and then wonder why the same class of failure returns. Those items are too vague to be verifiable. A good CDN action item has an owner, a measurable outcome, and a test that proves it works before you close it. Instead of "improve purge reliability," write "Purge queue drain alert must fire when depth >10k for >2m and must be tested by injecting 20k purge jobs in staging." I've seen teams cut repeat incidents by more than half just by making this change in how they write follow-ups. Each item should directly map to a specific root cause or amplifying factor you identified, not just generic hygiene.

We also categorize fixes by their leverage on recurrence, because not all fixes are equal. Quick mitigations like better alerts are necessary but they don't stop the next poisoned object from being cached. Structural fixes like semantic config validation or tiered cache isolation do. I track this for every postmortem and require at least one structural fix per major incident, even if it takes longer to ship. Without that discipline, you'll accumulate a backlog of alert tweaks while the underlying architecture stays fragile. Prioritize fixes that make the entire class of failure impossible, not just the exact replay of the last incident. That's how you build real resilience instead of playing whack-a-mole with symptoms.

# Example: OPA Rego policy enforcing postmortem action item
# Postmortem #2024-11: "Require max-age validation & purge rate limiting"
package cdn.config

deny[msg] {
  input.cache_control.max_age < 60
  msg := sprintf("max-age %v is too low; minimum is 60s per postmortem #2024-11", [input.cache_control.max_age])
}

deny[msg] {
  not input.purge.rate_limit.requests_per_minute
  msg := "purge endpoint must define rate_limit.requests_per_minute"
}

deny[msg] {
  input.purge.rate_limit.requests_per_minute > 1000
  msg := sprintf("purge rate limit %v exceeds global max of 1000/min", [input.purge.rate_limit.requests_per_minute])
}

# CI pipeline gate: python -m opa check --policy cdn_policy.rego cdn.json