SRE Practices for CDN: Applying Site Reliability Engineering to Content Delivery

Back in 2019 we shipped what looked like a harmless cache-key normalization at 2 p.m. on a Tuesday. Within twenty minutes our Frankfurt edge hit ratio fell from 94% to 31%, our origins started shedding connections, and checkout pages timed out for one of our largest retail customers. That was the afternoon our team stopped treating CDN operations like a traditional NOC and started applying site reliability engineering for real. If you've run a CDN at any serious scale, you've felt that same drop in your stomach.

Why Traditional CDN Operations Can't Keep Up With Edge Scale

For years, most CDN teams operated on a ticket-and-alert model. You'd get paged for origin 5xx spikes, you'd manually purge, you'd SSH into a PoP and restart ATS or Varnish, and you'd write a post-mortem. That works when you have twenty PoPs and a few hundred customers. It doesn't work when you have 250+ PoPs, thousands of configs deployed daily, and traffic that can double in minutes because a streaming event went viral. I've watched teams burn out trying to keep up with that volume by just hiring more ops engineers. You don't scale that way.

The core tension with SRE CDN work is that CDNs look deceptively simple from the outside - cache static assets, proxy dynamic traffic - but the failure modes are brutal. A bad regex in a VCL, a stale TLS cert, an inconsistent purge queue, and suddenly you don't have one broken server, you have a globally replicated bad decision. Traditional ops focuses on mean time to repair. SRE practices force you to focus on mean time to detect bad changes before they replicate, and on building systems that don't need a human to flip a switch at 3 a.m. That shift from reactive to engineered reliability is what separates teams that constantly fight fires from teams that actually control their error budgets.

That doesn't mean you throw away everything your network operations centre learned. You codify it. We started by taking our top five most frequent pages - cache invalidation delays, origin health flaps, certificate renewal failures, config deployment drift, and log delivery lag - and asking which ones required human judgment and which were just undifferentiated work. You'd be surprised how many were the latter. Our work on CDN Toil Reduction Through Automation: Eliminating Repetitive Operations Work came directly from that audit, but we also had to be careful not to automate ourselves into a new class of outage, which is exactly The CDN Automation Reliability Paradox: When Automation Causes More Outages that every senior team hits around year two of their automation journey.

# SLO burn-rate alerting for CDN edge availability
# Based on 99.95% availability SLO over 28 days
groups:
- name: cdn-slo-burn
  rules:
  - alert: CDNHighBurnRate
    expr: |
      (
        sum(rate(cdn_edge_requests_failed_total[5m]))
        /
        sum(rate(cdn_edge_requests_total[5m]))
      ) > (0.0005 * 14.4) # 14.4x burn for 2% budget in 1 hour
    for: 5m
    labels:
      severity: critical
      tier: edge
    annotations:
      summary: "CDN edge burn rate is 14.4x - error budget will exhaust in ~2 hours"
      runbook: "https://runbooks.internal/cdn/burn-rate"

  - alert: CDNLowBurnRate
    expr: |
      (
        sum(rate(cdn_edge_requests_failed_total[1h]))
        /
        sum(rate(cdn_edge_requests_total[1h]))
      ) > (0.0005 * 6) # 6x burn - slower, needs investigation
    for: 15m
    labels:
      severity: warning
    annotations:
      summary: "CDN edge burn rate is 6x - investigate origin and cache layer"

Defining Service Level Objectives That Match How Users Experience the CDN

A lot of teams copy-paste availability SLOs from their origin services and call it CDN reliability. That's a mistake. Users don't care if your edge daemons are up; they care if their content is fresh, fast, and actually served. When we rebuilt our SLOs, we threw out simple "edge uptime" and replaced it with four service level indicators that map to real pain: successful request rate as measured at the edge, cache hit ratio for cacheable content, time to first byte at the 95th percentile from real user monitoring, and origin offload failure rate. If you're not measuring from the client perspective, you're measuring the wrong thing. The web.dev Articles on Core Web Vitals and the HTTP Working Group Specifications on caching semantics were surprisingly useful references when we were arguing about what "successful" even meant - does a stale-while-revalidate hit count as a success? Does a 304 count?

Here's what I learned the hard way: don't set your SLO target based on what marketing wants to promise. Set it based on what you can actually measure and what your current system delivers. We collected 90 days of RUM and edge log data before we set anything. Our edge request success was hovering around 99.97% already, so promising 99.95% gave us a realistic 21 minutes of error budget per 28 days without lying to customers. But our P95 TTFB was 180ms globally, so we couldn't in good conscience set an SLO of 100ms - we'd burn that budget every day in Southeast Asia. We set it at 220ms initially, then used performance work to pull it down to 140ms over two quarters. That honesty builds trust with product teams, because they know when you say you're out of budget, you mean it.

You also need to treat your SLOs as code. If your SLO definitions live in a Google Doc, they don't exist. We store ours as versioned configs that generate both dashboards and alerting rules, so there's no drift between what we report and what pages us. That approach lines up closely with proper CDN Error Budget Management: Balancing Reliability with Feature Velocity, and it's saved us more than once when someone wanted to "temporarily" lower a threshold to ship a feature.

#!/usr/bin/env python3
# safe_config_rollout.py - Canary CDN config to 5% of PoPs with automatic rollback
import time
from cdn_api import deploy_config, get_error_rate, rollback

CANARY_POPS = ["fra02", "iad03", "sin01"]
ERROR_THRESHOLD = 0.002  # 0.2% error rate
EVALUATION_WINDOW = 300  # 5 minutes

def canary_deploy(config_version):
    print(f"Deploying config {config_version} to canary PoPs: {CANARY_POPS}")
    deploy_config(version=config_version, pops=CANARY_POPS, strategy="canary")

    time.sleep(EVALUATION_WINDOW)

    for pop in CANARY_POPS:
        rate = get_error_rate(pop=pop, window="5m")
        print(f"[{pop}] error rate: {rate:.4f}")
        if rate > ERROR_THRESHOLD:
            print(f"Threshold breached in {pop}, rolling back globally")
            rollback(version=config_version)
            return False

    print("Canary healthy, proceeding to full fleet rollout")
    deploy_config(version=config_version, pops="all", strategy="staged")
    return True

if __name__ == "__main__":
    canary_deploy(config_version="v2024.11.42")

Error Budgets and Toil Budgets That Keep CDN Teams Honest

An error budget isn't just a number you put on a dashboard. It's a decision-making tool. When we introduced ours, the biggest pushback wasn't from engineering, it was from product. Suddenly, "we've burned 80% of our budget this month, so we need to freeze non-critical purges and config changes" wasn't an engineering preference, it was policy. That policy only works if everyone agrees on it before an incident. We run a monthly review where we look at budget burn, classify it - was it a bad deployment, an origin issue, an upstream provider, or just normal noise? - and then decide if we need to adjust velocity or invest in reliability work. Without that ritual, site reliability engineering just becomes a new title for the same old on-call rotation.

We pair error budgets with toil budgets, something I don't see enough CDN teams talk about. We cap toil - manual, repetitive, automatable work - at 30% of a team's time, max 40% in a bad week. If you're spending more than that hand-holding cache purges, rotating certs manually, or babysitting log pipelines, you aren't doing SRE work, you're doing ops with a fancier name. We track toil in our sprint board with a specific label, and if we hit 35% for two sprints in a row, we stop feature work and automate the biggest toil source. That discipline is how we cut our manual purge-related pages from 12 per week to less than one per month.

# Prometheus SLO burn-rate alert for CDN edge latency
# SLO: 99.9% of requests serve TTFB < 120ms over 28-day window
groups:
- name: cdn-slo-latency
  rules:
  - record: slo:cdn_ttfb_burn_rate_5m
    expr: |
      (
        sum(rate(cdn_edge_requests_total{ttfb_ms=">120"}[5m]))
        /
        sum(rate(cdn_edge_requests_total[5m]))
      ) / (1 - 0.999)

  - alert: CDNHighBurnRate
    expr: slo:cdn_ttfb_burn_rate_5m > 14.4
    for: 5m
    labels:
      severity: critical
      burn_rate: "14x"
    annotations:
      summary: "CDN latency SLO burning too fast (14x) - auto-rollback canary"
      runbook: "/runbooks/cdn-latency-burn"