Circuit Breaker Implementation at CDN Edge: Preventing Cascade Failures

Back in 2019, we pushed a tiny origin timeout change — 200ms to 500ms — thinking we were being generous. Thirty minutes later, half our North American edge PoPs were shedding traffic and origins were drowning in retries. It wasn't the origins that failed us; we failed to stop the edge from hammering them. That night taught me that CDN fault tolerance isn't about making origins invincible, it's about teaching the edge when to stop asking.

When Retries Become the Outage: Anatomy of an Edge-Origin Cascade

Most engineers picture a cascade failure as an origin going dark, but at the edge it usually starts while the origin is still technically alive. You’ve got 300 PoPs, each with dozens of servers, all seeing elevated TTFB at the same time. Without a circuit breaker, every cache miss spawns a retry, then a revalidation, then a thundering herd on the next MISS. I’ve watched a single slow API endpoint — 800ms p95 instead of the usual 90ms — turn into 40k RPS of origin load because our edge logic was “helpfully” retrying on 502 and 504. We didn’t have an origin outage, we manufactured one.

The math is brutal and predictable. If your edge concurrency is 10k requests per second and your origin starts shedding 30% of traffic with timeouts, a naive retry policy with just one retry effectively pushes 13k attempts per second at an origin that can barely handle 7k. Add in stale-while-revalidate or collapsed forwarding that breaks, and you’re not just amplifying load — you’re synchronizing it. That’s why cascade prevention has to live on the data plane, not in some central control plane that learns about the problem 30 seconds too late. We learned to treat this as an SRE Practices for CDN: Applying Site Reliability Engineering to Content Delivery problem: define failure narrowly, detect it locally, and act within milliseconds.

What you’re really building is edge resilience against partial degradation. Don’t wait for HTTP 500s. Timeouts, TCP connection failures, TLS handshake errors, and even 429s from an overwhelmed origin are all signals that should count toward your breaker’s failure budget. I now count any origin response that takes longer than our p99 SLO as a half-failure, because latency is often the earliest sign of collapse. The HTTP semantics around retryability and cacheability documented in the HTTP Working Group Specifications are useful here — they’ll remind you why blindly retrying idempotent GETs isn’t free when you’ve got a global fleet doing it in lockstep.

// Edge circuit breaker - per-origin state in shared memory (Rust/Wasm edge worker)
struct EdgeBreaker {
    state: AtomicU8, // 0=Closed, 1=Open, 2=HalfOpen
    failures: SlidingWindowCounter, // 10s window, 100 buckets
    success_probes: AtomicU8,
}

impl EdgeBreaker {
    fn should_allow(&self) -> bool {
        match self.state.load(Ordering::Relaxed) {
            0 => true, // Closed - all traffic flows
            1 => {
                // Open - check if cool-down elapsed (e.g., 30s)
                if self.cooldown_expired() {
                    self.state.compare_exchange(1, 2, Ordering::SeqCst, Ordering::Relaxed).ok();
                    true // Allow one probe into Half-Open
                } else {
                    false // Fast-fail: serve stale or 503 + Retry-After
                }
            },
            2 => self.success_probes.load(Ordering::Relaxed) < 3, // Half-Open: limited probes
            _ => false
        }
    }

    fn record(&self, result: OriginResult) {
        // Timeouts and 5xx count as 1.0, p99 latency breaches as 0.5
        let weight = match result {
            OriginResult::Timeout | OriginResult::Status5xx => 1.0,
            OriginResult::Slow(p99_breach) if p99_breach => 0.5,
            _ => 0.0
        };
        self.failures.add(weight);
        self.evaluate_threshold(); // e.g., 50% failure rate over 10s -> Open
    }
}

Building a Three-State Breaker That Actually Works at the Edge

Forget the textbook circuit breaker you’d use inside a microservice. At the CDN edge you can’t afford coordination, you can’t store much state, and you must make a decision in under a millisecond. We run a classic three-state machine — Closed, Open, Half-Open — but we shard it per PoP, per origin hostname, and sometimes per URL pattern. A checkout API gets a far more aggressive threshold than a static image origin. That scoping is the difference between protecting one origin and accidentally tripping the breaker for everything behind it.

In Closed state, we’re counting failures in a sliding window rather than a simple counter. I’ve moved away from fixed 60-second windows; they’re too slow to trip and too slow to recover. Our sweet spot is a 10-second rolling window with 100 buckets, rolling every 100ms. If the weighted failure rate crosses 50% and we’ve seen at least 200 requests in that window, we open. The request minimum prevents a single PoP with low traffic from flapping open on three stray 503s. When we do open, we don’t just return errors — we serve stale from cache if we have it, or synthesize a 503 with a precise Retry-After: 15 and stale-if-error semantics that browsers and downstreams respect, which aligns with guidance you’ll find in web.dev Articles on resilient loading patterns.

Half-Open is where most edge implementations get lazy and break. Don’t just flip back to Closed after a timer expires. We allow exactly three probe requests through after a 30-second cool-down, and they must be staggered — one every 500ms. If two of three succeed within our latency SLO, we close. If any probe times out, we reopen immediately with an exponential backoff: 30s, 60s, 120s, capped at 5 minutes. That jitter and backoff is critical; without it, 200 PoPs all probing at the same second will create the very cascade you were trying to prevent. This probe logic ties directly into how we think about CDN Error Budget Management: Balancing Reliability with Feature Velocity — every false probe burns budget we’d rather spend serving stale gracefully.

-- OpenResty / NGINX edge Lua: fast-fail before origin fetch
local breaker = require "edge.breaker"
local origin_key = "api.origin.example.com:/v2/checkout"

local state = breaker.get_state(origin_key)

if state == "OPEN" then
  -- Don't touch the origin. Serve stale or fail fast.
  local stale = cache.lookup_stale(ngx.var.request_uri, 300) -- 5m stale window
  if stale then
    ngx.header["X-Cache-Status"] = "STALE-CIRCUIT-OPEN"
    ngx.header["Retry-After"] = "30"
    return stale:serve()
  end
  ngx.status = 503
  ngx.header["Retry-After"] = "30"
  ngx.say("Origin circuit open - edge shedding load")
  return ngx.exit(503)
end

if state == "HALF_OPEN" and not breaker.allow_probe(origin_key) then
  -- Another worker is already probing, shed this request
  return serve_stale_or_503()
end

-- Proceed to origin, record result asynchronously
local res, err = fetch_origin()
breaker.record_result(origin_key, {
  status = res and res.status or 0,
  latency_ms = res and res.ttfb or 5000,
  is_timeout = (err == "timeout")
})

Tuning Thresholds With Real Production Signals

There’s no universal threshold that works for every origin, and copying defaults from a library will hurt you. For our edge cache fleet, we ended up with very different settings for dynamic APIs versus static origins, and we re-tune them quarterly. The biggest mistake I see teams make is setting failure thresholds too high — “we’ll open at 80% errors so we don’t flap.” By the time you hit 80% 5xx at the edge, your origin has been on fire for a full window and you’ve already amplified the failure. We trip far earlier on latency, and we bias toward opening quickly and recovering cautiously.

We also learned to track success differently. It’s not enough to count 200 OKs. A 200 that took 4 seconds is a failure for an edge SLO where we promise 150ms TTFB. That’s why our breaker records weighted failures: full weight for timeouts and 5xx, half weight for responses over our p99 latency objective. And we never run breakers in isolation from observability — every state transition emits a counter and a trace event so we can see when a PoP is flapping. If you’re still doing manual threshold changes via tickets, you’re burning hours you shouldn’t be; we automated rollbacks tied to error budget burn, a pattern we documented in our CDN Toil Reduction Through Automation: Eliminating Repetitive Operations Work work.

Below is what we actually run in production today for two origin classes. These aren’t theoretical — they’re from a 90-day window across ~180 PoPs serving ~12 Tbps peak. Notice how the dynamic API breaker is far more sensitive and has a shorter half-open probe budget, because the cost of hammering a stateful checkout origin is much higher than revalidating a static asset.

-- OpenResty / Varnish Lua - PoP-local circuit breaker
local dict = ngx.shared.circuit_state
local WINDOW = 10  -- seconds
local MIN_REQ = 30
local ERROR_THRESHOLD = 0.5
local OPEN_TTL = 30

local function should_allow(origin_key)
  local state = dict:get(origin_key .. ":state")
  if state == "OPEN" then
    -- Half-open probe: allow 1 request per second to test recovery
    local probe_lock = dict:add(origin_key .. ":probe", 1, 1)
    if probe_lock then return true end -- this request is the probe
    return false, "circuit_open"
  end
  return true
end

local function record_result(origin_key, is_error)
  local bucket = math.floor(ngx.now())
  local key = origin_key .. ":b:" .. bucket
  dict:incr(key, is_error and 1 or 0, 0, 11)
  dict:incr(origin_key .. ":total:" .. bucket, 1, 0, 11)

  -- aggregate last 10 buckets
  local errors, total = 0, 0
  for i=0, WINDOW-1 do
    errors = errors + (dict:get(origin_key .. ":b:" .. (bucket - i)) or 0)
    total = total + (dict:get(origin_key .. ":total:" .. (bucket - i)) or 0)
  end

  if total >= MIN_REQ and (errors / total) >= ERROR_THRESHOLD then
    dict:set(origin_key .. ":state", "OPEN", OPEN_TTL)
    ngx.log(ngx.WARN, "breaker OPEN for ", origin_key, " err=", errors, "/", total)
  elseif state == "OPEN" and not is_error then
    -- successful probe closes the circuit
    dict:set(origin_key .. ":state", "CLOSED")
  end
end