The Circuit Breaker Pattern for CDN Origins: Preventing Cascade Failures

Back in 2019, I watched a single struggling origin database take down an entire global CDN rollout in under four minutes. It wasn't a DDoS and it wasn't a traffic spike — it was our own edge nodes doing what they were told to do: retry relentlessly. Every 502 turned into three more requests, and those retries synchronized across 200 PoPs until the origin couldn't even serve health checks. That's the moment I stopped treating circuit breakers as optional.

When Retries Turn a Blip Into a Full Outage

Most engineers think cascade failures start with a big bang. In my experience, they start with a whimper — a slow origin, a deploy that bumps p99 latency from 80ms to 600ms, a downstream API that starts shedding 5% of requests. Without a breaker, the CDN does exactly the wrong thing. It holds connections open longer, exhausts your Origin Connection Pooling Optimization: Reducing TCP Overhead at CDN Scale limits, and then fans out retries to alternate origins that are already sharing the same sick database. I've seen this amplify a 5% error rate into a 100% outage faster than any alerting system can page you.

The math is brutal because the edge is distributed. Say you've got 150 PoPs, each allowing 2 retries with a 1-second timeout. One origin instance hiccups and 30% of PoPs see a timeout simultaneously. That's not 30% more load — it's a synchronized thundering herd that hits the moment the origin is most vulnerable. Standard retry logic with exponential backoff helps a single client, but it doesn't protect a shared resource. The CDN needs a collective memory, a way to say "we've seen enough failures, we need to stop calling for a while" — that's the circuit breaker — and it has to be coordinated or at least statistically consistent across the fleet.

We learned this the hard way after coupling naive retries with passive health checks. Our passive checks were marking nodes down after three consecutive 502s, but the retry logic was still firing before the check could trip. The breaker has to sit in front of the retry logic, not behind it. If you let the request through to the origin just to count it as a failure, you've already done the damage. The patterns described in the HTTP Working Group Specifications for 502, 503, and 504 handling make it clear these status codes are signals to shed load, not invitations to retry harder.

// Naive retry vs. breaker-guarded origin fetch (Edge Worker pseudo-code)
async function fetchOriginNaive(request, origin) {
  for (let i = 0; i < 3; i++) {
    try {
      // This will hammer a failing origin on every PoP
      return await fetch(origin, { signal: AbortSignal.timeout(1000) });
    } catch (e) { if (i === 2) throw e; }
  }
}

// With circuit breaker - fail fast at the edge
async function fetchOriginWithBreaker(request, origin) {
  const breaker = getBreakerForOrigin(origin.hostname);
  if (breaker.isOpen()) {
    // Don't even open a socket. Serve stale or error immediately.
    return serveStaleOr503(request);
  }
  try {
    const res = await fetch(origin, { signal: AbortSignal.timeout(1000) });
    if (res.status >= 500) throw new Error(`origin ${res.status}`);
    breaker.recordSuccess();
    return res;
  } catch (e) {
    breaker.recordFailure();
    throw e;
  }
}

Implementing Closed, Open, and Half-Open States at the Edge

There are only three states, but getting them right at CDN scale is tricky. In the Closed state, everything flows normally and we're just counting failures in a rolling window. Once the failure rate trips the threshold, we flip to Open — that's where we fail fast without touching the origin at all for a cooldown period. I prefer returning a 503 with a Retry-After and serving stale if we have it, rather than queuing. After the cooldown, we don't just slam the door open again; we enter Half-Open and let a tiny probe volume through to test the waters. If those probes succeed, we close the circuit. If they don't, we go straight back to Open.

Where you store that state matters more than the state machine itself. You can't rely on a single global Redis lock — you'll add latency to every origin request and create a new single point of failure. We've had much better results with local, per-PoP breakers using a shared sliding window counter that syncs via gossip or eventually consistent broadcasts. Each PoP makes its own decision, but they converge quickly because they're all seeing similar error rates. This also plays nicely with Origin Health Monitoring Patterns: Active Probes, Passive Checks, and Hybrid Approaches; your active health prober can be the sole client allowed through during Half-Open, so real user traffic isn't used as a canary.

Don't make the mistake of using a simple failure count. A count of 50 failures means nothing if you've served 1M successes in the same window. You want a failure ratio over a minimum volume, plus a slow-call detector. An origin that times out 40% of requests over 10 seconds is far more dangerous than one that returns five 500s in a minute during a deploy. I also separate error types — 502/504 and timeouts trip the breaker fast, while 503 with a proper Retry-After header might get a longer grace period, as recommended in web.dev Articles on resilient fetching.

// Production-ready breaker state with rolling window (Go-style for edge)
type OriginBreaker struct {
  mu sync.RWMutex
  state State // CLOSED, OPEN, HALF_OPEN
  window *SlidingWindow // 10s window, 1s buckets
  openedAt time.Time
}

func (b *OriginBreaker) Allow() bool {
  b.mu.RLock()
  defer b.mu.RUnlock()
  switch b.state {
  case OPEN:
    if time.Since(b.openedAt) > 30*time.Second {
      // Transition to HALF_OPEN is handled by a timer, not here
      return false // still shed load until prober succeeds
    }
    return false
  case HALF_OPEN:
    // Only allow 1 probe per second per PoP
    return b.window.ProbeTokens().Allow(1)
  default: // CLOSED
    return true
  }
}

func (b *OriginBreaker) Record(res *OriginResponse) {
  b.window.Add(res.Success)
  failureRate := b.window.FailureRate()
  total := b.window.TotalCount()

  if b.state == CLOSED && total > 100 && failureRate > 0.5 {
    b.tripOpen()
  } else if b.state == HALF_OPEN && failureRate < 0.1 {
    b.closeCircuit()
  }
}

Calibrating Thresholds That Actually Protect Your Origin

Default thresholds will hurt you. I've walked into setups where the breaker was set to open after 50% failures over 60 seconds with a 5-second cooldown. That's an eternity when your origin's thread pool is exhausted. By the time the breaker opened, the origin was already OOMKilled and the cooldown was too short to let it recover before the herd returned. You have to tune for your origin's actual recovery profile, not a textbook example. For a typical API origin running behind an auto-scaling group, you want short windows and aggressive shedding.

We run load tests that intentionally brown-out the origin — we inject 200ms of extra latency and 20% 500s — and then watch how the breaker reacts. You're looking for two things: time-to-open under 5 seconds from the start of the brown-out, and zero retries sent while Open. If your breaker takes 20 seconds to open, you've missed the window. This is where it ties directly into Origin Retry Strategies for CDN Resilience: Backoff, Jitter, and Budget Limits; your retry budget should be zero when the breaker is Open. There's no point in adding jittered backoff if the circuit is already telling you to stop.

The table below is what we've settled on after three years of tuning for a high-throughput e-commerce API origin that serves about 40k RPS through the CDN. It handles traffic for product pages where stale-while-revalidate is acceptable, so we can be aggressive. If your origin serves non-cacheable POSTs, you might lengthen the Open timeout slightly, but don't go past 60 seconds — you want fast feedback.

Wiring the Breaker Into Your Edge Logic: A Worker-Based Implementation

The theory's clean, but the implementation is where most teams trip. Don't put circuit breaker logic on your origin — by then it's already too late. The breaker has to live at the edge, in front of the origin, where it can shed load before it ever leaves the PoP. For most of us that means a Cloudflare Worker, Fastly Compute, or an Nginx/Lua layer sitting in your shield PoP. We've had the best luck implementing it as a lightweight Worker that wraps every origin fetch.

You'll need shared state, and that's the tricky part. A single PoP tracking its own failures isn't enough — 10 failures in Ashburn might be noise, but 10 failures per PoP across 200 PoPs is a 2,000-request stampede. We use a centralized counter in Redis or Workers KV with a 10-second sliding window. Each edge increments a failure counter on 5xx, timeouts, and connection errors — not on 4xx, those are client problems. When the window hits something like 50% failures over at least 20 requests, we flip the breaker to Open and set a TTL of 30 seconds. That TTL is your cool-down. Every edge checks that single key before it even attempts a TCP connect, so you're not wasting Origin Timeout Configuration Guide: Connect, Read, and Total Timeout Tuning budgets on doomed requests.

What you do while you're Open matters just as much. Don't just return a 503. If you've got stale content, serve it — even 5-minute-old stale is better than an error page for most GETs. For API routes where stale won't cut it, return a synthetic 503 with a Retry-After: 30 header and a fallback JSON payload your frontend can handle gracefully. We've also learned to short-circuit hedging while tripped. If you're using Origin Request Hedging: Sending Parallel Requests to Cut Tail Latency, you must disable that secondary hedged request when the breaker is Open — otherwise you're doubling the load you're trying to prevent.

// Cloudflare Worker - Edge Circuit Breaker (simplified)
const STATE_KEY = "cb:origin:api.example.com";
const FAILURE_THRESHOLD = 0.5; // 50% failure rate
const MIN_REQUESTS = 20;
const OPEN_TTL_SECONDS = 30;
const HALF_OPEN_PROBES = 3;

async function fetchWithBreaker(request, env) {
  const state = await env.CIRCUIT_KV.get(STATE_KEY, "json") || { status: "closed", failures: 0, total: 0 };

  // 1. If OPEN, fail fast and serve stale
  if (state.status === "open") {
    const stale = await caches.default.match(request);
    if (stale) return stale;
    return new Response(JSON.stringify({ error: "Origin unavailable, try again" }), 
      { status: 503, headers: { "Retry-After": "30", "X-Circuit-State": "open" } });
  }

  // 2. If HALF-OPEN, allow only a probe request
  if (state.status === "half-open" && state.probes >= HALF_OPEN_PROBES) {
    return new Response("Circuit half-open: probe limit reached", { status: 503, headers: { "X-Circuit-State": "half-open" } });
  }

  try {
    const originRes = await fetch(request, { cf: { cacheTtl: 60 } });
    if (originRes.status >= 500) throw new Error(`Origin 5xx: ${originRes.status}`);
    
    // Success - reset on half-open, decay counter on closed
    await env.CIRCUIT_KV.put(STATE_KEY, JSON.stringify({ status: "closed", failures: 0, total: 0 }));
    return originRes;
  } catch (err) {
    const newState = await incrementFailure(state);
    // Trip if window breached
    if (newState.total >= MIN_REQUESTS && (newState.failures / newState.total) >= FAILURE_THRESHOLD) {
      await env.CIRCUIT_KV.put(STATE_KEY, JSON.stringify({ status: "open", openedAt: Date.now() }), { expirationTtl: OPEN_TTL_SECONDS });
    } else {
      await env.CIRCUIT_KV.put(STATE_KEY, JSON.stringify(newState), { expirationTtl: 10 });
    }
    throw err;
  }
}

Advanced Patterns and Debugging Flapping Breakers

Once your basic breaker is stable, you'll want to add bulkheading. Don't use one global breaker for your entire origin. We run separate breakers per origin hostname *and* per critical path — api.example.com/checkout gets its own breaker from api.example.com/catalog. That way a failing checkout service doesn't take down your product images. We've seen a single slow database query on /search trip a global breaker and knock out the whole site, when 95% of origins were perfectly healthy. Per-route breakers take a bit more KV namespace, but they've saved us from at least three major incidents this year.

The most common operational headache is flapping — the breaker rapidly cycling Open -> Half-Open -> Closed -> Open. This almost always means your Half-Open probe is too aggressive or your success threshold is too low. Don't mark the circuit Closed after a single successful probe. We require 3 consecutive successful probes spaced 2 seconds apart with p95 origin latency under 800ms before we close it. Add jitter to that probe interval so 200 PoPs don't all slam the origin at the exact same millisecond when the TTL expires. And make sure your failure detection distinguishes real origin failures from edge TLS issues — a bad cert rotation can look like an origin outage if you're not careful. We explicitly exclude Origin SSL Verification at CDN Edges: Preventing Man-in-the-Middle Attacks failures from the breaker count and alert on them separately.

For observability, you absolutely need four metrics on a dashboard: state (0=closed, 1=open, 2=half-open), failure rate over the window, requests short-circuited per second, and origin p95 latency during half-open. If you're seeing a high short-circuit rate but the origin looks healthy in your APM, your threshold is too sensitive. We tune ours so the breaker trips maybe once or twice a quarter on real incidents — if it's tripping weekly, you're not protecting the origin, you're just creating your own outage. Set a PagerDuty alert for any transition to Open, and log the X-Circuit-State header on every response so you can correlate with user complaints.

Frequently Asked Questions