CDN Incident Response: Playbooks and Recovery
CDN Incident Categories
| Category | Severity | Examples |
|---|---|---|
| Provider outage | Critical | CDN provider global/regional outage |
| Cache poisoning | Critical | Wrong content served to users |
| DDoS attack | High | Volumetric or application-layer attack |
| Origin failure | High | Origin unreachable, CDN serving stale content |
| Certificate issue | High | Expired or misconfigured SSL certificate |
| Performance degradation | Medium | Elevated latency, reduced cache hits |
Playbook: CDN Provider Outage
- Detect — Monitoring alerts on elevated error rates or latency
- Confirm — Check CDN provider status page, test from multiple locations
- Communicate — Notify stakeholders: "CDN provider experiencing issues, we are monitoring"
- Failover — If multi-CDN is configured, shift traffic to backup CDN via DNS
- Bypass — If no backup CDN, update DNS to point directly to origin (accept higher latency)
- Monitor recovery — Watch provider status page, test periodically
- Restore — When provider recovers, shift traffic back gradually
- Post-mortem — Document timeline, impact, and improvement actions
Playbook: Cache Poisoning
Cache poisoning means wrong content is cached and served to users. This is critical because it affects all users until the cache is purged.
| Metric | Edge Processing | Origin Processing | Improvement |
|---|---|---|---|
| Latency (p50) | 8–15 ms | 120–300 ms | 15–20x faster |
| Cache Hit Ratio | 85–95% | N/A | Reduces origin load |
| TLS Handshake | 1–5 ms (nearby PoP) | 50–150 ms | 10–30x faster |
| Bandwidth Cost | $0.01–0.04/GB | $0.05–0.12/GB | 3–5x cheaper |
| Availability | 99.99% (multi-PoP) | 99.9% (single DC) | 10x fewer outages |
- Identify — Determine which URLs are affected
- Purge immediately — Purge all affected URLs
- Block the vector — Identify how poisoning occurred (typically through unkeyed headers or query params)
- Verify — Confirm correct content is now served
- Prevent — Review cache key configuration to include relevant headers
Playbook: DDoS Attack
- Detect — Traffic spike alerts, error rate increase
- Classify — Volumetric (bandwidth), protocol (SYN flood), or application (HTTP flood)
- Engage CDN mitigation — Most CDNs auto-mitigate; enable "Under Attack" mode if available
- Apply rate limits — Tighten rate limiting rules for attack traffic patterns
- Update WAF rules — Block attack signatures
- Monitor — Track attack evolution and adjust mitigations
Playbook: Origin Failure
When the origin server becomes unreachable:
- Stale content serving — CDNs with stale-while-revalidate or stale-if-error continue serving cached content
- Custom error page — Configure a branded error page instead of CDN's default
- Origin failover — If multiple origins are configured, CDN fails over automatically
- Status communication — Serve a cached status page informing users of the issue
Communication Template
[INVESTIGATING] We are investigating elevated error rates.
[IDENTIFIED] The issue has been identified as [X].
[MONITORING] A fix has been implemented. We are monitoring.
[RESOLVED] The incident has been resolved. Duration: X minutes.
Affected: [regions/services]. Root cause: [summary].Post-Incident Review
Every CDN incident should produce: timeline with timestamps, root cause analysis, user impact assessment, detection time analysis (how long before alerts fired?), and action items with owners and deadlines.