CDN Runbook Automation: Converting Manual Procedures into Executable Scripts

It was 2:17 AM when the alert fired — edge latency spiking across our Chicago POP and cache hit ratio dropping through the floor. I pulled up our “Cache Node Failure” runbook, a 47-step wiki page last updated eight months earlier, and by step 12 I realized half the CLI commands had changed after our last traffic manager migration. We fixed it that night, but it took 43 minutes of frantic copy-paste and guesswork that should’ve taken five. That incident is why I stopped treating runbooks as documentation and started treating them as code.

Why Manual Runbooks Crack Under Real Incident Pressure

Most CDN runbooks start with good intentions. Someone writes down exactly what they did to fix a problem, adds screenshots, maybe links to a Grafana dashboard, and saves it to Confluence. On a quiet Tuesday afternoon, that document looks perfect. Under incident pressure, it falls apart. I’ve watched engineers — smart, experienced ones — skip steps, paste a command into the wrong terminal, or misread a cache status because they were juggling three Slack threads while trying to parse a wall of text. The cognitive load is just too high when you’re trying to protect origin load and edge availability at the same time.

Manual procedures also rot incredibly fast in CDN environments. Edge configurations, purge APIs, health check endpoints, and routing policies change constantly. A command that worked last quarter might now return a deprecated warning or, worse, silently do nothing. When we audited our runbooks two years ago, we found that 38% of the shell commands referenced flags that no longer existed and 22% pointed to dashboards that had been decommissioned. That’s not an outlier; that’s normal if your runbooks aren’t tested regularly. If you’re looking at broader reliability practices, this is where ideas from SRE Practices for CDN: Applying Site Reliability Engineering to Content Delivery really start to matter — you can’t treat docs as static when the system underneath never stops moving.

Runbook automation doesn’t mean you remove the human. It means you convert that fragile wiki page into executable procedures that actually do the checks, handle the edge cases, and fail safely if something’s wrong. A good first test I use is simple: can a junior on-call run this at 3 AM without having to interpret what the author meant? If the answer is no, it’s not automated enough. Here’s a pattern I’ve come to rely on for wrapping a manual check into something scriptable, using a simple validation wrapper instead of trusting human eyes:

#!/bin/bash
# executable-check.sh - Replace "eyeball the dashboard" with a verifiable check
set -euo pipefail

POP="ord-03"
THRESHOLD=85

# Fetch real-time hit ratio from CDN monitoring API
HIT_RATIO=$(curl -s -H "Authorization: Bearer $CDN_API_TOKEN" \
  "https://monitor.internal/v1/pops/$POP/hit-ratio" | jq -r '.hit_ratio')

echo "[INFO] POP $POP hit ratio: ${HIT_RATIO}%"

# Automated decision - no human interpretation required
if (( $(echo "$HIT_RATIO < $THRESHOLD" | bc -l) )); then
  echo "[ALERT] Hit ratio below ${THRESHOLD}% - triggering diagnostic collection"
  ./collect-edge-diagnostics.sh --pop "$POP" --output "/tmp/diag-$POP-$(date +%s).tar.gz"
  exit 2
else
  echo "[OK] Hit ratio healthy"
  exit 0
fi

From Wiki Page to Executable Procedure: A Practical Conversion

Converting a manual runbook isn’t about rewriting everything at once. I’ve made that mistake — trying to automate a 50-step failover procedure in one go and ending up with a brittle 800-line script nobody trusted. What works better is picking one procedure you run often and converting it step-by-step. For us, it was the stale cache purge verification. The manual version said: “1. Run purge, 2. Check 5 edge nodes with curl, 3. Verify Age header, 4. If not purged, repeat.” Every engineer did step 2 slightly differently, and nobody consistently checked the Age and X-Cache-Status headers the right way.

The trick is to make each step idempotent and observable. Your CDN scripts should be safe to run twice, should log what they did, and should never assume a previous step succeeded. I always add explicit pre-flight checks — do I have the right credentials, is the API reachable, is the target POP actually in service? — before touching any production traffic. You’ll also want to tie your scripts to how you think about risk. We started gating automated purges and route changes against our error budgets, a practice we borrowed after reading CDN Error Budget Management: Balancing Reliability with Feature Velocity. If we’ve already burned too much budget for the week, the script refuses to do a non-critical purge and tells you why. It’s a small guardrail that’s saved us from turning a minor issue into a major one more than once.

For anything that touches HTTP caching behavior, I always validate against what the spec actually says, not just what we think our CDN does. The HTTP Working Group Specifications on cache-control and the practical guidance in web.dev Articles are my go-to references when I’m automating header checks. That way your script isn’t just automating the old tribal knowledge — it’s correcting it. Here’s a compact Python example we use to automate purge verification across multiple edge nodes without manual curl loops:

#!/usr/bin/env python3
# verify_purge.py - Executable runbook step for purge validation
import requests
import sys

EDGE_NODES = ["edge-ord03-a", "edge-ord03-b", "edge-ord03-c"]
URL_TO_CHECK = "https://cdn.example.com/assets/app.v123.js"
EXPECTED_CACHE_STATUS = "MISS" # After purge, first fetch should be MISS

def check_node(node):
    headers = {"Host": "cdn.example.com", "Pragma": "no-cache"}
    # Force request through specific edge via internal routing header
    try:
        resp = requests.get(URL_TO_CHECK, headers={**headers, "X-Target-Edge": node}, timeout=5)
        cache_status = resp.headers.get("X-Cache-Status", "UNKNOWN")
        age = resp.headers.get("Age", "none")
        print(f"{node}: Status={resp.status_code} X-Cache={cache_status} Age={age}")
        return cache_status == EXPECTED_CACHE_STATUS and resp.status_code == 200
    except requests.RequestException as e:
        print(f"{node}: FAILED - {e}")
        return False

results = {node: check_node(node) for node in EDGE_NODES}
if all(results.values()):
    print("Purge verified on all edges")
    sys.exit(0)
else:
    failed = [k for k, v in results.items() if not v]
    print(f"Purge incomplete on: {', '.join(failed)}")
    sys.exit(1)

Building Automated Response That’s Safe to Run Without You

The biggest fear teams have about runbook automation is obvious: what if the automation makes it worse? I’ve felt that too. Early in my career I wrote an auto-failover script that dutifully drained an entire POP because a single synthetic monitor flapped. We learned the hard way that automated response needs the same rigor as any other production code — reviews, tests, and circuit breakers. Every executable procedure we ship now has three non-negotiables: a dry-run mode that shows what it would do, an automatic abort if origin health drops during execution, and a clear audit log of who or what triggered it.

Start by automating detection and diagnostics before you automate remediation. It’s a much safer place to build confidence. Instead of jumping straight to “automatically reroute traffic,” start with scripts that collect the right logs, run the right curl checks with the correct headers, and present a recommendation: “POP ord-03 is serving 42% MISS with 2s origin latency, recommendation: drain.” That alone cuts mean time to acknowledge by minutes and reduces the toil that burns out on-call teams. As we’ve scaled this, the reduction in repetitive work has been the most visible win — a theme covered well in CDN Toil Reduction Through Automation: Eliminating Repetitive Operations Work — because engineers stop spending their shifts doing manual health checks and start reviewing automation that already did them.

We track every manual vs. automated procedure with real operational metrics, not gut feel. If you can’t measure the improvement, you can’t justify expanding automation to riskier runbooks. The table below shows what we saw after converting our top five most-used CDN runbooks into executable scripts over a six-month period. The numbers aren’t theoretical; they’re from our PagerDuty and Jenkins logs, and they’re why our on-call rotation finally stopped dreading cache-related pages.

#!/usr/bin/env python3
# cdn_runbook_automated_purge_failover.py --dry-run --key "product-api-v2" --pop ord,ash,lax
import argparse, sys, time, requests

def check_origin_health(primary_origin, timeout=5):
    try:
        r = requests.get(f"https://{primary_origin}/healthz", timeout=timeout)
        return r.status_code == 200 and r.elapsed.total_seconds() < 1.5
    except requests.RequestException:
        return False

def execute_runbook(cache_key, primary, secondary, dry_run=False):
    is_healthy = check_origin_health(primary)
    print(f"[PRE-CHECK] Primary origin {primary} healthy: {is_healthy}")

    if not is_healthy:
        action = f"Would failover {primary} -> {secondary} and purge key: {cache_key}"
        if dry_run:
            print(f"[DRY-RUN] {action}")
            return True
        print(f"[ACTION] Purging key {cache_key} and failing over...")
        # purge_cache_key(cache_key)  # API call to CDN provider
        # failover_origin(primary, secondary)
        time.sleep(3)
        # Post-check: verify from 3 edge PoPs
        for pop in ["ord", "ash", "lax"]:
            status = requests.get(f"https://cdn.example.com/{cache_key}", 
                                  headers={"X-Debug-POP": pop}).status_code
            assert status == 200, f"POP {pop} still failing with {status}"
        print("[VERIFY] All 3 PoPs returning 200. Failover successful.")
        return True
    print("[SKIP] Origin healthy, no action needed. Idempotent exit.")
    return True

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--key", required=True)
    parser.add_argument("--dry-run", action="store_true")
    args = parser.parse_args()
    execute_runbook(args.key, "origin-primary.example.com", "origin-backup.example.com", args.dry_run)