Origin Rate Limiting from CDN Edges: Protecting Backends from Overload

Back in 2019, we pushed a 50% off sale for a major retail client and watched their origin crumble in 90 seconds flat. Edges were healthy, cache hit ratio looked fine at 94%, but that remaining 6% of misses turned into 80,000 requests per second hammering a backend that could handle 8,000. It wasn't a DDoS — it was our own cache cold start. That's when I learned origin rate limiting isn't optional; it's the seatbelt you don't think about until you're already crashing.

Why Your Origin Melts Before Your Edges Do

CDN edges are built to scale horizontally. You’ve got hundreds of PoPs, each with thousands of connections handling traffic without breaking a sweat. Your origin, on the other hand, is often a single autoscaling group, a Kubernetes cluster, or — in the worst cases I’ve seen — a single rack of servers that someone swore would “scale when we need it.” It doesn’t. The math simply doesn’t work in your favor. If you have 300 edge nodes and each one forwards just 30 cache misses per second, you’re already pushing 9,000 RPS to an origin that starts timing out at 5,000. And that’s on a quiet day.

What makes this tricky is that origin overload rarely looks like overload at first. It looks like slightly higher latency, a few more 502s, maybe a bump in database CPU. I’ve spent too many nights chasing “random” origin timeouts only to realize our edges were doing exactly what they were supposed to do — fetching fresh content — but doing it all at once. This couples tightly with how you manage connections; if you haven’t tuned your edge-to-origin path, read up on Origin Connection Pooling Optimization: Reducing TCP Overhead at CDN Scale because an inefficient pool can amplify a small miss storm into a full-blown connection flood. Add retries to the mix and you’ve got a feedback loop where failed fetches are retried aggressively, which is why you need sane Origin Retry Strategies for CDN Resilience: Backoff, Jitter, and Budget Limits alongside any rate limiting.

You can’t fix this with bigger origins alone. You have to control the flow at the edge, where the requests originate. The goal isn’t to drop user traffic — it’s to queue, shape, and shed origin-bound requests so the backend stays responsive for the requests that do get through. Standards around how responses signal retry behavior are well documented in the HTTP Working Group Specifications, particularly around 429 and 503 handling.

# Token bucket for origin-bound requests per PoP
# Local limiter: 200 req/s to origin, burst 400
origin_rate_limiter {
  type token_bucket
  rate 200
  burst 400
  key origin_hostname # per-origin bucket, not per-client
  action {
    on_limit queue timeout=2s max_queue=1000
    on_queue_full return 503 retry_after=1
  }
  # Critical: don't count cache HITs, only MISS/PASS
  count_only statuses = ["miss", "pass", "revalidate"]
}

# Global view via control plane (eventual consistency)
global_sync {
  interval 1s
  datastore redis_cluster
  aggregate_limit 8000 # hard cap across all edges
}

Designing a Distributed Limiter That Doesn't Lie to You

The simplest approach is a local limiter on each edge node — say, 100 requests per second per server to the origin. It’s easy to deploy and doesn’t need coordination. I’ve used this pattern for years on smaller properties and it works fine until you hit about 50+ nodes. Then the math gets fuzzy. A local limit of 100 RPS across 200 servers is still 20,000 RPS globally, which might be double what your origin can take. You’ve limited the per-node blast, but you haven’t actually protected the backend. Local limits are a seatbelt, not a full braking system.

That’s why we moved to a two-tier model: a strict local token bucket for instant protection, plus a global distributed counter for precision. The global limiter doesn’t need to be perfectly consistent — we can tolerate a second or two of drift. We use a sliding window counter in Redis or our control plane that aggregates origin-bound requests across all PoPs. Each edge checks its local bucket first (zero latency decision), then checks its allocated share of the global budget. If the global budget is exhausted, edges start queuing or serving stale. It’s not perfect, but it’s cut our origin 5xx spikes by about 80% during miss storms. This pairs directly with how you detect failure — your Origin Health Monitoring Patterns: Active Probes, Passive Checks, and Hybrid Approaches should inform how quickly you tighten limits when origins start degrading.

// Adaptive origin limiter logic (pseudo-Go)
// Runs on each edge, synced via control plane every 500ms

type OriginLimiter struct {
  localBucket  *TokenBucket // 150 RPS, burst 300
  globalQuota  int          // allocated from global counter
  originRTT    time.Duration
  errorRate    float64
}

func (l *OriginLimiter) Allow(req *Request) Decision {
  if !l.localBucket.Take(1) {
    return QueueOrStale(req, "local_limit")
  }

  // Adaptive throttling: if origin latency > 800ms or 5xx > 1%
  // cut quota by 50% and serve stale
  if l.originRTT > 800*time.Millisecond || l.errorRate > 0.01 {
    l.globalQuota = l.globalQuota / 2
    if l.globalQuota < 10 {
      return ServeStale(req)
    }
  }

  if l.globalQuota <= 0 {
    // Don't return 503 to user if we have stale
    if hasStale(req) {
      return ServeStale(req)
    }
    return Return503WithRetryAfter(2)
  }

  l.globalQuota--
  return ForwardToOrigin(req)
}

Tuning Thresholds Without Punishing Real Users

Setting the number is the hardest part, and I’ll be honest — we got it wrong for a long time. We started with a static limit based on load test results, like “origin handles 6,000 RPS, so set limit to 5,500.” It worked until traffic patterns shifted and our most critical product API needed burst capacity while image thumbnails could wait. A single global number treats all origin traffic as equal, which it never is. You need to segment limits by origin route or host. Your `/api/checkout` origin might need a tight 500 RPS limit with a small burst, while your image origin can handle 10,000 RPS with a huge queue because it’s fronted by an async image processor.

The best approach I’ve found is starting with your origin’s measured p95 capacity under normal load, not its theoretical max. Run a sustained load test and find where latency starts to climb — not where it fails. If latency jumps at 7,000 RPS, set your steady-state limit at 5,000-5,500 and allow bursts up to 7,000 for 5-10 seconds. Then add adaptive concurrency based on origin signals like queue depth, latency, and error rate. When the origin’s p95 latency doubles or your 5xx rate crosses 0.5%, the limiter should automatically back off instead of waiting for a human to edit a config. The web.dev Articles on optimizing time-to-first-byte have solid guidance on how origin queuing directly impacts user-perceived performance, which helps justify these trade-offs to product teams.

Don’t set and forget. We review our origin limit metrics weekly, and we alert on limiter engagement. If a limiter is queueing more than 5% of origin requests for more than 2 minutes, that’s not a limiter problem — it’s a capacity or cache-key design problem. Similarly, if you’ve never hit your limiter in a month, it’s probably too generous.

From Theory to Config: Building a Shield-Aware Limiter That Actually Holds

Most teams screw this up by enforcing limits at every edge PoP independently. If you've got 40 PoPs and you set a 200 req/s limit per PoP, your origin isn't protected at 200 req/s — it's exposed to 8,000 req/s the moment a cache-busting attack or a bad deploy hits. We've seen it melt a perfectly healthy Kubernetes origin in under 12 seconds. Don't do per-PoP fixed windows. Push your rate limiting decision to your shield tier or origin-adjacent aggregator where you can see the global picture.

The pattern that works reliably at scale is a centralized token bucket with a local, stale fallback. Your shield layer checks a fast counter (Redis, Garnet, or even a shared memory atomic if you're on a single shield PoP) for the origin's global budget — say, 3,500 req/s with a burst of 600 — and only proxies if a token is available. Each edge PoP also keeps a tiny local bucket, maybe 25 req/s with a 50 burst, that's only used when the central counter is unreachable. That way you don't hard-fail just because Redis hiccupped for 800ms. You’ll want to emit two metrics here at minimum: origin_limiter_allowed vs origin_limiter_delayed per PoP, sampled every 5s. If you can’t see which PoP is eating your budget, you can’t tune it.

Here’s a stripped-down Lua example for OpenResty / NGINX at the shield that we run in production. It uses a sliding window via lua-resty-limit-req backed by a shared dict, but in a real distributed setup you’d swap the dict for a Redis INCR + PEXPIRE call. Note how we fail open and set a soft queue instead of instantly 503ing — you want to shed load gracefully, not thundering-herd your own retries:

-- shield/limiter.lua - Origin protection at the shield tier
local limit_req = require "resty.limit.req"
-- 3500 req/s global, burst 600, with Redis fallback
-- local_dict is only for fail-open mode
local lim, err = limit_req.new("origin_shield_dict", 3500, 600)

if not lim then
  ngx.log(ngx.ERR, "failed to init limiter: ", err)
  return ngx.exec("@proxy_to_origin") -- fail open
end

local key = "origin:api.example.com"
-- Try central Redis counter first (pseudo-code)
local tokens, err = redis:eval([[
  local c = redis.call('INCR', KEYS[1])
  if c == 1 then redis.call('PEXPIRE', KEYS[1], 1000) end
  return c
]], 1, key)

if not tokens or tokens > 3500 then
  -- No token: serve stale or 429 with Retry-After
  ngx.header["Retry-After"] = "2"
  ngx.header["X-Origin-Limiter"] = "exhausted; tokens=" .. (tokens or 0)
  return ngx.exit(429)
end

-- Token acquired, proxy with hedged timeout
ngx.header["X-Origin-Limiter"] = "passed"
return ngx.exec("@proxy_to_origin")

Advanced Patterns and Why Your Limiter Is Still Misbehaving

Once the basics are in place, you’ll run into the interaction between rate limiting, collapsing, and hedging. If you’re already using Origin Request Hedging: Sending Parallel Requests to Cut Tail Latency to cut your p99 from 1.2s to 180ms, your limiter needs to count *hedged* requests separately. We count the first request against the primary budget (3,500 req/s) and hedged retries against a smaller secondary budget (350 req/s, ~10%). If you don’t, a single slow origin response can double your effective load exactly when you’re trying to protect it. The other gotcha is request collapsing — make sure your limiter runs *after* collapsing in the stack. There’s no point burning a token for 47 identical concurrent requests for /api/config.json that should have been collapsed into one origin fetch.

When you’re debugging, don’t just look at 429s. The real signal is in the timing of 503s and origin queue depth. A healthy limiter shows a flat origin concurrency line — say, 850-900 concurrent connections — even while edge ingress spikes to 25k req/s. A broken limiter shows sawtooth patterns: origin concurrency plummets to 120, then spikes to 2,400, then timeouts explode. Check three things in order: 1) Are your Origin Timeout Configuration Guide: Connect, Read, and Total Timeout Tuning values too tight? A 2s read timeout with a 1.8s p95 origin will create retry storms that look like limiter failures. 2) Is your shield failing closed? Run redis-cli --latency during an incident — if p99 Redis latency is >45ms, your fail-open local bucket isn't kicking in. 3) Are you accidentally sharing a limiter key between your API origin and your image origin? We've seen that take down checkout because image optimization ate the budget.

For high-stakes origins, switch from static thresholds to adaptive limiting. Instead of a hard 3,500 req/s, have your origin return a X-Origin-Load: 0.85 header or a 1-byte /healthz that reports EWMA load every 500ms. Your shield then adjusts the token refill rate with a simple AIMD loop: if load >0.8, multiply limit by 0.9; if load <0.6 for 30s, multiply by 1.05. It’s the same idea as TCP congestion control and it prevents you from having to guess capacity after every autoscale event. And please, don’t forget Origin SSL Verification at CDN Edges: Preventing Man-in-the-Middle Attacks still has to happen *before* you count the request — you don’t want to spend your precious origin budget proxying to an attacker with a spoofed origin IP that failed TLS verification.

Frequently Asked Questions