CDN Back-Pressure Mechanisms: Signaling Overload Before Systems Break

Back in 2019, I watched a 40-PoP CDN fold in under 90 seconds during a flash sale. It wasn't a capacity problem — we had headroom on paper. It was a back-pressure problem. Every edge kept accepting requests, queuing them, retrying origins that were already drowning, and amplifying the load until nothing could recover. I've seen that same pattern repeat a half-dozen times since, and it always comes down to this: if your CDN can't say "not right now" gracefully, it'll be forced to say nothing at all.

Why CDN Edges Fail Without Explicit Back-Pressure

Most CDN architectures are optimistically designed. You expect cache hits to absorb 85-95% of traffic, origins to respond in under 200ms, and connections to recycle cleanly. That works until it doesn't. When an origin slows to 2-3 seconds or a cache tier goes degraded, edges don't naturally push back. They hold connections open, spawn more threads, and queue requests in memory until the kernel starts killing processes or load balancers mark the node as dead. By then you've lost the ability to shed load intelligently.

I've learned to treat back-pressure as a first-class signal path, not an error case. Your edge should know its own saturation point before the OS does, and it should communicate that state upstream and downstream in a way clients and peers can act on. That means explicit concurrency limits, queue depth thresholds, and standardized overload headers — not just hoping timeouts will save you. If you're building your reliability model around SRE Practices for CDN: Applying Site Reliability Engineering to Content Delivery, this is where those practices become concrete. Without back-pressure, your SLI burn rate becomes non-linear and your error budgets evaporate in minutes, which is exactly what CDN Error Budget Management: Balancing Reliability with Feature Velocity warns against when you lack load shedding controls.

# Envoy-style overload manager + circuit breaker for an edge proxy
# This enforces back-pressure before the host OOMs - tune for your hardware

overload_manager:
  resource_monitors:
  - name: "envoy.resource_monitors.fixed_heap"
    typed_config:
      "@type": type.googleapis.com/envoy.config.resource_monitor.fixed_heap.v2alpha.FixedHeapConfig
      max_heap_size_bytes: 8589934592 # 8GB
      max_heap_bytes_percent: 0.85

  actions:
  - name: "envoy.overload_actions.stop_accepting_requests"
    triggers:
    - name: "envoy.resource_monitors.fixed_heap"
      threshold:
        value: 0.92
  - name: "envoy.overload_actions.shed_p99_latency"
    triggers:
    - name: "envoy.resource_monitors.fixed_heap"
      threshold:
        value: 0.80

circuit_breakers:
  thresholds:
  - priority: DEFAULT
    max_connections: 60000
    max_pending_requests: 4000
    max_requests: 20000
    max_retries: 3
    retry_budget:
      budget_percent:
        value: 15
      min_retry_concurrency: 5

Signaling Overload So Upstreams Actually Back Off

Shedding load locally is only half the job. If your shield tier or origin still thinks the edge is healthy, it will keep sending traffic that will be dropped. You need to signal overload explicitly and early. For HTTP, that means using 503 with a precise Retry-After, 429 when it's a tenant or client limit, and connection-level signals like HTTP/2 GOAWAY or HTTP/3 connection close before you hit hard errors. I prefer to send 503 + Retry-After: 2 plus a CDN-Capacity: degraded; q=0.12 style header for internal layers — it gives L7 load balancers and retry logic something deterministic to act on instead of guessing from latency.

Don't rely on clients to be well-behaved. According to the HTTP Working Group Specifications, Retry-After and 503 semantics are deliberately flexible, but many clients ignore them or retry immediately. That's why internal back-pressure between your own tiers matters more. We had a case where our edge-to-shield retries were amplifying a 3k RPS origin hiccup into 18k RPS. We fixed it by adding jittered exponential backoff, a shared token bucket for origin concurrency, and propagating the shield's queue depth back to edges via response headers. The web.dev Articles on retry strategies cover the client side well, but on the CDN side you own both client and server — so you can enforce the contract. Automation here is critical; as discussed in CDN Toil Reduction Through Automation: Eliminating Repetitive Operations Work, manual tuning of these thresholds doesn't scale across hundreds of PoPs.

// Go middleware: admission control with early 503 + Retry-After
// Returns degraded signal before queue builds

func BackPressureMiddleware(next http.Handler, limiter *AdmissionLimiter) http.Handler {
  return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    // Cheap checks first: concurrency and queue depth
    if limiter.Concurrency() > 18000 || limiter.QueueDepth() > 3500 {
      w.Header().Set("Retry-After", "2")
      w.Header().Set("CDN-Capacity", "shed; reason=concurrency; retry=2000ms")
      // Add Load header per draft - helps LBs fail fast
      w.Header().Set("CDN-Load", "load=92; limit=100")
      http.Error(w, "capacity degraded - retry shortly", http.StatusServiceUnavailable)
      metrics.Inc("edge.shed.concurrency")
      return
    }

    // Origin-aware shedding: if origin p95 > 800ms, start shedding low-priority
    if limiter.OriginP95() > 800 && r.Header.Get("Priority") == "low" {
      w.Header().Set("Retry-After", "5")
      w.Header().Set("CDN-Capacity", "shed; reason=origin_latency")
      http.Error(w, "origin pressure - shedding non-critical", http.StatusServiceUnavailable)
      metrics.Inc("edge.shed.origin_pressure")
      return
    }

    next.ServeHTTP(w, r)
  })
}

What We Measure Before We Start Dropping

Effective back-pressure isn't about a single CPU threshold. I track a small set of leading indicators that consistently fire 30-60 seconds before hard failure. Queue depth is the most honest — if your accept queue or pending request count keeps climbing while active connections are flat, you're not keeping up. Origin p95 and p99 tell you whether to shed before you cascade. And per-PoP 5xx rate plus retry amplification tells you if your shedding is actually reducing global load or just moving it.

We set thresholds per PoP class because a 10 Gbps edge with 32 cores behaves very differently from a 100 Gbps spine cache. The key is to alert and act on pressure, not just on failure. I run a 30-second rolling window for these signals; if two or more fire simultaneously, we enter degraded mode automatically and widen Retry-After values. That early action has cut our incident blast radius by about 70% compared to waiting for health check failures.

Implementing Back-Pressure at the Edge: Queues, Circuit Breakers, and Graceful 503s

You've got the signals — now you've got to enforce them before the kernel does it for you. The biggest mistake we see teams make is setting limits at 100% capacity. By the time your edge hits 100% CPU or your origin connection pool is exhausted, you're already dropping traffic randomly and latency's gone exponential. We run hard concurrency limits at about 80-85% of benchmarked max throughput. For a typical 32-core NGINX/Envoy edge, that's around 800-1200 active concurrent requests, not the 2,000 it can technically handle for 30 seconds before it falls over.

Don't just drop randomly when you hit that limit. That's how you nuke your hit ratio and punish clean cacheable traffic. We've learned to shed in priority order: first, kill background revalidation and prefetch, then defer stale-while-revalidate, then shed low-priority API traffic and bot tiers, and only then start rate-limiting user-facing GETs. In practice that means classifying requests at ingress — add a priority: 1-4 header — and your shedding logic respects it. It's the difference between a 2% error rate on low-value traffic and a 20% outage for everyone. If you're automating this shedding, be careful; poorly tuned auto-remediation flips you into flapping territory fast, which is exactly The CDN Automation Reliability Paradox: When Automation Causes More Outages we keep running into.

The signal itself matters as much as the drop. A silent TCP reset or timeout teaches the upstream nothing — it'll just retry harder. You need an explicit, retryable signal: 503 Service Unavailable with a Retry-After: 5 and ideally a Retry-After-Ms or X-Edge-Load: 92% header for smarter LBs. Envoy, Fastly, and CloudFront all respect 503+Retry-After for backing off if you've configured it, but ATS and Varnish upstreams won't unless you tell them to. Here's a pattern we use at the edge middleware layer to enforce a token-bucket concurrency gate with prioritized shedding:

// Go middleware for edge back-pressure with prioritized shedding
// Tuned for ~850 max concurrent reqs per instance, p95 origin latency 120ms
func BackPressure(next http.Handler) http.Handler {
    sem := make(chan struct{}, 850) // hard concurrency limit
    queue := make(chan *http.Request, 250) // short queue, not infinite

    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        priority := getPriority(r) // 1=critical (HIT), 4=background

        select {
        case sem <- struct{}{}:
            defer func() { <-sem }()
            // CoDel-style check: if queueing > 50ms, shed low priority
            if len(queue) > 150 && priority >= 3 {
                w.Header().Set("Retry-After", "5")
                w.Header().Set("X-Shed-Reason", "edge-concurrency-high")
                http.Error(w, "Edge at capacity", http.StatusServiceUnavailable)
                return
            }
            next.ServeHTTP(w, r)
        default:
            // At limit - immediately shed with explicit backoff
            w.Header().Set("Retry-After", "3")
            w.Header().Set("X-Shed-Reason", "concurrency-exhausted")
            // Track shed.count by priority for alerting
            metrics.Inc("edge.shed", priority)
            http.Error(w, "Overloaded", http.StatusServiceUnavailable)
        }
    })
}

Advanced Patterns: Adaptive Shedding and Troubleshooting Silent Overload

Static limits work until they don't — traffic mix changes, origin latency spikes from 40ms to 400ms, and suddenly your 850 concurrency limit is 5x too high. That's where adaptive patterns come in. We run a Vegas-style controller that watches p95 response time and load average, not just connection count. If p95 for origin fetches creeps past 180ms for 15 seconds, or the 1-minute load avg exceeds 0.85 * cores, we start shedding 10% of priority-4 traffic, then ramp to 50% if it doesn't recover in 30 seconds. It's essentially CoDel for CDNs: you drop based on sojourn time, not queue length. This keeps latency bounded at ~200ms p95 instead of letting it balloon to 2-3 seconds before you react.

Troubleshooting back-pressure is tricky because the failure is often silent from the client's view. If your upstreams ignore Retry-After — and many default Kubernetes or NGINX upstreams do — you haven't created back-pressure, you've created a retry storm. We've seen a single edge shedding 500 RPS turn into 2,500 RPS of retries from three L7 LBs. Check two things immediately: first, are your upstreams configured with exponential backoff + jitter and a circuit breaker (Envoy's max_retries: 2 and circuit_breakers: max_pending_requests 250)? Second, are you emitting the right cardinality of metrics — edge.shed.count by PoP, priority, and reason — so you can actually see shedding happening before users report 503s? If your dashboards only show 5xx rate, you won't know if it's origin failure or intentional shedding.

Finally, don't try to debug this ad-hoc during an incident. Back-pressure tuning is exactly where manual tribal knowledge fails. You need versioned, executable checks: "if shed rate > 100/sec for 2m, auto-lower concurrency limit by 15%" should be a tested script, not a Slack message. That's why we converted all our manual load-shedding procedures into CDN Runbook Automation: Converting Manual Procedures into Executable Scripts — so the on-call doesn't have to remember the right curler command at 3am. And after every shedding event, run a proper review; our best tuning insights came from CDN Postmortem Analysis: Learning from Incidents to Prevent Recurrence, where we realized our 429s were being retried instantly while 503s weren't.

Frequently Asked Questions