CDN Incident Management Runbook: Detection, Triage, and Resolution

A CDN failure is never just a performance dip — it is a global user-facing outage. When edge nodes serve stale content, TLS handshakes fail, or cache poisoning spreads in seconds, every second of confusion adds to MTTR. This runbook gives DevOps and SRE teams a prescriptive, battle-tested workflow for detection, triage, and resolution.

1. Detection: Know Before Your Users Do

Effective detection relies on layered observability. Relying on a single uptime check from one region guarantees blind spots. A mature CDN monitoring stack correlates four signal types: synthetic monitoring, Real User Monitoring (RUM), edge logs, and origin health.

Your alerting should be tied to Service Level Indicators (SLIs) that reflect user experience, not just infrastructure health. For a deeper dive into which metrics matter, review our guide to CDN monitoring and observability and the core CDN performance metrics every team should track.

Key SLIs and Alert Thresholds

Configure alerts on burn rate, not absolute values, to avoid alert fatigue. Example thresholds for a 99.9% availability SLO:

• Availability: Alert if 5xx rate > 0.5% for 5 minutes OR > 1% for 2 minutes at any PoP.
• Latency: Alert if p95 TTFB > 800ms for 10 minutes or p95 cache HIT latency increases > 2x baseline.
• Cache Efficiency: Alert if global cache hit ratio drops > 15% below 7-day rolling average.
• Origin Health: Alert if origin 5xx + 429 exceeds 2% or origin latency p95 > 1500ms.

Instrument both edge and origin independently. A synthetic check that bypasses the CDN (direct-to-origin via hosts override) is critical for isolation.

# Prometheus alerting rule for CDN edge errors
groups:
  - name: cdn.slo
    rules:
      - alert: CDNHigh5xxBurnRate
        expr: |
          sum(rate(cdn_edge_responses_total{status=~"5.."}[5m]))
          /
          sum(rate(cdn_edge_responses_total[5m])) > 0.005
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "CDN 5xx burn rate exceeded 0.5% for 5m"
          runbook: "https://flyfishnwa.com/cdn-incident-management-runbook#triage"

Detection Checklist (First 3 Minutes)

1. Acknowledge alert in PagerDuty/Opsgenie — stop escalation.
2. Open incident channel (#inc-cdn-YYYY-MM-DD) and create status page draft.
3. Validate alert: check 2+ PoPs, RUM, and curl -I from external vantage point.
4. Assign roles: Incident Commander (IC), Communications Lead, and Ops Lead.

2. Triage: Classify, Isolate, and Communicate

Triage is about answering three questions fast: How bad is it? How wide is it? Is it CDN or origin? Use a severity matrix to force consistent decisions under pressure.

Isolation Framework

Run these three tests in parallel to isolate the layer:

• Edge HIT vs MISS: curl -I -H "Pragma: no-cache" vs normal request. If only HITs fail, you have cache poisoning or bad cached object. If both fail, suspect config, WAF, or origin.
• PoP Comparison: Test via curl --resolve or vendor's PoP-specific hostnames. If only one PoP fails, drain it via load balancing. See CDN load balancing algorithms for draining strategies.
• Direct Origin Test: Bypass CDN entirely. If origin succeeds, fault is at edge. If origin fails, focus on autoscaling, database, or upstream. Understanding this split requires a solid grasp of your CDN architecture overview.

During triage, freeze non-emergency deploys and config changes. Post initial status update within 10 minutes for SEV1/SEV2: “Investigating elevated 5xx in NA-East, 12% error rate, mitigation in progress. Next update in 15 min.”

3. Resolution: Mitigation Playbooks

Have pre-approved, reversible mitigations. The IC should be able to execute without waiting for change approval. Document the exact command, expected impact, and rollback for each.

Playbook A: Bad Cache / Poisoned Object

Symptoms: HITs return 500/502, MISSes succeed, hit ratio normal but error ratio high. Often caused by origin briefly sending bad headers (e.g., Cache-Control: max-age=86400 on an error page).

# Fastly / Cloudflare / Akamai purge examples - choose one
# Cloudflare: purge by URL (preferred over purge-all)
curl -X POST "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/purge_cache" \
  -H "Authorization: Bearer $CF_TOKEN" \
  -H "Content-Type: application/json" \
  --data '{"files":["https://www.example.com/checkout/*"]}'

# Verify purge propagation
for pop in iad ord lhr nrt; do
  curl -sI -H "CF-IPCountry: US" https://www.example.com/checkout/ | grep -i "cf-cache-status"
done

Mitigation: Purge by surrogate key or URL prefix, not full purge-all unless global poisoning confirmed. Set Cache-Control: no-store at origin temporarily to prevent re-caching bad response. For image or asset issues, also consider purging the CDN image optimization tier separately.

Playbook B: Configuration / WAF / TLS Failure

Symptoms: All PoPs return 403/503, WAF challenge loop, or TLS handshake failures after a recent rule or cert deployment. Check vendor status page and recent audit log.

• Rollback last config via versioned config: cdn config rollback --to v142 --reason "INC-421 WAF FP"
• If WAF rule is blocking, disable rule ID in monitor mode first, then disable. Do not disable entire WAF unless under active DDoS attack mitigation is failing open.
• For TLS: Re-issue and deploy backup certificate, or temporarily enable vendor-managed cert.

Playbook C: Origin Overload After Cache Bypass

Symptoms: Origin latency spikes, 502 Bad Gateway from edge, autoscaling lag. Often triggered by a purge-all or failover.

• Enable stale-while-revalidate and stale-if-error at edge: serve stale for 60-300s while origin recovers.
• Rate-limit origin fetches: enable origin shield and request collapsing.
• Increase origin concurrency and scale horizontally before re-enabling traffic.

Playbook D: PoP or Vendor Outage

Symptoms: Single region latency/errors, vendor status page confirms incident. Mitigation: Drain PoP via DNS or load balancer weighting, or fail over to secondary CDN if multi-CDN is configured. For multi-CDN, shift traffic via 20% increments with health checks.

4. Communication and Post-Incident Learning

Incident communication is not optional. Assign a Communications Lead who is not the IC. Update internal stakeholders every 15 minutes for SEV1, every 30 minutes for SEV2, and customers via status page every 30 minutes. Use a consistent template: Impact, Scope, Mitigation, ETA.

Once resolved, schedule a blameless postmortem within 48 hours. The goal is not to assign fault but to identify systemic fixes. A strong postmortem answers:

• What was the timeline from first signal to detection (MTTD) and to mitigation (MTTR)?
• Why did detection not happen earlier? Was alert threshold too loose or signal missing?
• What allowed the bad config/cache to propagate globally without canary?
• What is the one architectural change that would prevent recurrence (e.g., config canary, surrogate-key purging, origin shield)?

Convert every action item into a ticket with an owner and SLO (e.g., “Add canary deployment for CDN config — Owner: Platform Team — Due: 14 days”). Track completion in your incident review meeting. Update this runbook with learnings — a runbook that never changes is a runbook that is not being used.

Finally, run game days. Simulate a poisoned cache or PoP failure quarterly. Test your purge scripts, rollback permissions, and multi-CDN failover when no customers are impacted, so they work when they are.

Last updated: September 10, 2026 · Reviewed by FlyFish NWA SRE Team · Next review: December 10, 2026