Origin Connection Draining for Maintenance: Graceful Backend Transitions

We had a scheduled maintenance window at 2 AM to patch three origins behind a busy product catalog. The plan was simple: pull one host at a time, let the CDN shift traffic, patch, and bring it back. Ten seconds after we removed the first host from the load balancer, our error rate spiked to 1.2% and we saw a burst of 502s that shouldn't have happened. The connections we'd thought were idle weren't, and the cutover was anything but graceful.

Why Pulling an Origin Abruptly Still Drops In-Flight Requests

When your CDN maintains persistent connections to origins, those connections live in a pool for a long time. That's great for performance — you avoid TLS and TCP handshakes on every request — but it creates a hidden coupling during a backend transition. If you just remove an origin's IP from DNS or mark it down in the load balancer, the edge still has a handful of warm TCP connections sitting there. I've seen pools hold 40-60 connections per edge PoP to a single origin, some idle, some mid-stream. Kill the origin process or firewall the port and those in-flight requests get a TCP RST or a timeout. The client sees a 502 or 504 even though you have two other perfectly healthy origins right next to it.

That's where connection draining comes in, and it's fundamentally different from just failing health checks. A health check tells the edge not to open *new* connections. Draining tells it to stop using *existing* ones gracefully while letting current requests finish. Without that distinction, you're racing the edge's connection reuse logic. We learned this the hard way when we relied solely on passive health checks. Our Origin Connection Pooling Optimization: Reducing TCP Overhead at CDN Scale setup was highly tuned for reuse, which meant connections lingered for 90 seconds by default. During deploys, that reuse was working against us.

The other subtle failure is half-closed connections. The origin might send a FIN, but if the CDN has already dispatched a request on that connection a millisecond earlier, you get a race condition. The HTTP spec doesn't save you here unless both sides cooperate. As described in the HTTP Working Group Specifications, a server that wants to close gracefully should signal it won't accept new streams or requests, rather than just closing the socket. For HTTP/1.1 that means sending Connection: close, for HTTP/2 it means a GOAWAY frame. If you don't implement that signal, you'll always lose a few requests during any maintenance window, no matter how careful your orchestration is.

# Example: NGINX origin draining before shutdown
# Step 1 - Start failing health checks but keep serving traffic
# In your health check endpoint:
location /healthz {
    # During draining, return 503 to signal "don't open new connections"
    if (-f /var/run/draining) {
        return 503 "draining - do not route new requests";
    }
    return 200 "ok";
}

# Step 2 - Graceful worker shutdown
# nginx.conf
worker_shutdown_timeout 45s;  # Allow in-flight requests to finish
keepalive_timeout 75s;
keepalive_requests 1000;

# Deployment script
touch /var/run/draining
sleep 5  # Give CDN time to see 503s and stop pooling
nginx -s quit  # Graceful shutdown, not stop (which is immediate)

Coordinating the Draining Handshake Between CDN and Origin

A good graceful shutdown isn't just an origin concern; it's a handshake. The origin has to announce it's going away, and the CDN has to respect that announcement quickly. For years we tried to do this with only active health probes every 5 seconds. It wasn't fast enough. By the time the edge noticed the origin was unhealthy, it had already sent another 20-30 requests down pooled connections that were about to be killed. We now use a two-signal approach: an explicit application signal plus a load balancer state change. On the origin, we trap SIGTERM and immediately start returning Connection: close on HTTP/1.1 or GOAWAY on H2, while also flipping the health check to 503.

On the CDN side, you need to make sure your origin selection logic actually pools *per health state*. Some older configs we audited kept using an already-established connection even after the health check failed, because the pool was keyed only by IP. You want the edge to drain its own pool the moment it sees that 503 or that close header. This ties directly into how you monitor origins. If you're using hybrid health checks as described in Origin Health Monitoring Patterns: Active Probes, Passive Checks, and Hybrid Approaches, you can treat a single 503 on the health endpoint as a trigger to set the pool's draining flag, rather than waiting for three consecutive failures. That shaves 10-15 seconds off your backend transition time, which matters when you have a tight maintenance window.

Don't forget your retry budget either. Even with perfect draining, you'll have that tiny window where a request was already dispatched just as the drain started. Without safe retries, that request still fails. We wrap our origin fetches with a limited, idempotency-aware retry as outlined in Origin Retry Strategies for CDN Resilience: Backoff, Jitter, and Budget Limits — only for safe methods like GET and HEAD, and only once per edge try, with jitter to avoid thundering herds. The web.dev Articles on connection management also emphasize that clients (and CDNs are just fancy clients) should be prepared to retry on a closed persistent connection. That retry is your safety net, not your primary plan.

// Go origin with SIGTERM draining - cooperative shutdown
package main

import (
    "context"
    "net/http"
    "os"
    "os/signal"
    "syscall"
    "time"
)

func main() {
    isDraining := false
    mux := http.NewServeMux()
    
    mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
        if isDraining {
            http.Error(w, "draining", http.StatusServiceUnavailable)
            return
        }
        w.Write([]byte("ok"))
    })

    server := &http.Server{
        Addr:         ":8080",
        Handler:      mux,
        ReadTimeout:  10 * time.Second,
        WriteTimeout: 30 * time.Second,
        IdleTimeout:  75 * time.Second,
    }

    go func() {
        sig := make(chan os.Signal, 1)
        signal.Notify(sig, syscall.SIGTERM, syscall.SIGINT)
        <-sig
        isDraining = true // Flip health check immediately
        // Give CDN 8s to observe 503s and drain its pool
        time.Sleep(8 * time.Second)
        ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
        defer cancel()
        server.Shutdown(ctx) // Gracefully closes listeners, waits for handlers
    }()
    
    server.ListenAndServe()
}

Tuning Drain Duration and Pool Limits for Real Production Traffic

How long should you drain? Too short and you still cut off long-running requests — we had catalog image transforms that legitimately took 12-18 seconds. Too long and your maintenance window drags out, and you hold capacity hostage while you're trying to patch. I don't use a fixed number anymore. I look at our origin's p95 and p99 response times and our keepalive idle timeout. If your p99 is 850ms and your keepalive timeout is 60s, a 30-second drain is usually enough to let 99.9% of requests complete while still letting the CDN evict idle connections naturally. For origins that serve large file downloads or video segments, you might need 90 to 120 seconds.

You also need to tune concurrency during the drain. When you mark one origin as draining, the other two suddenly absorb its share. If you've got 3000 RPS across three origins and you drain one, the remaining two jump from 1000 to 1500 RPS each instantly. If their connection pools aren't sized for that burst, you'll get queueing and origin latency spikes that look like another outage. We now pre-warm the remaining origins by temporarily raising their max pool size and reducing their keepalive timeout for 60 seconds before we start the drain. It's a small trick, but it smoothed out our latency curves dramatically during rolling deploys.

The numbers below come from a test we ran last quarter on a fleet handling about 8k RPS at peak. We measured three different drain durations during identical backend transitions, with the same 45-second worker_shutdown_timeout. The impact on error rate and tail latency was not subtle, and it convinced our platform team to make 30 seconds the minimum standard for any stateful maintenance window.

Putting Drain Logic Into Practice: CDN and Origin Configs

Knowing how draining should work is one thing — wiring it up so it actually triggers reliably is where most teams slip. You’ve got two sides to coordinate, and if either one’s misconfigured you’ll still see 502s during deploys. On the CDN side, you want an explicit “draining” state for the origin, not just a failed health check. On the origin side, you need a graceful shutdown that stops accepting new keep-alive reuse while letting workers finish what they’ve got.

We typically run a 90-second drain window for standard API origins and push it to 300 seconds for origins handling large file uploads or WebSockets. That gives us headroom over our 60-second Origin Timeout Configuration Guide: Connect, Read, and Total Timeout Tuning total timeout without holding capacity hostage. If you set your drain timeout shorter than your origin’s read timeout, you’ve guaranteed hard drops — I’ve seen teams set a 30-second drain with a 60-second read timeout and then wonder why 0.8% of requests fail on every rolling restart. Align them: drain_timeout = read_timeout + 15-30s buffer. And don’t forget to verify your TLS handshakes still pass during drain — a failing Origin SSL Verification at CDN Edges: Preventing Man-in-the-Middle Attacks check will yank the origin out of rotation before draining even starts.

# 1. CDN control plane: Mark origin as draining (stop new assignments)
curl -X PATCH https://api.cdn.example/v1/pools/prod-api/origins/10.0.1.23 \
  -H "Authorization: Bearer $CDN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "state": "draining",
    "drain_timeout_sec": 90,
    "fail_open": false
  }'

# 2. Origin side: Nginx graceful shutdown
# nginx.conf
worker_shutdown_timeout 95s; # Must be > drain_timeout_sec

upstream_keepalive_timeout 60s;
keepalive_requests 1000;

# Signal Nginx to stop accepting, finish in-flight requests
nginx -s quit
# Or for systemd-managed origins:
# systemctl reload nginx  # with updated upstream weights

# 3. Health endpoint should advertise drain state
# GET /healthz -> 200 OK when active
# GET /healthz -> 503 + "Draining: 42 active conns" when draining
# CDN stops routing NEW requests on 503, but respects existing connections

Advanced Patterns and Troubleshooting Stuck Drains

Once you’ve got the basics, you’ll hit the edge cases that break naive draining. Long-lived connections are the biggest culprit. A standard 90-second drain won’t touch a WebSocket that’s been open for 45 minutes or an HLS segment download that’s multiplexed over HTTP/2. You can’t just wait forever — you need a two-phase drain. Phase one: stop new requests and send Connection: close on keep-alive responses for 30 seconds. Phase two: for any connections still active after the drain window, send a GOAWAY (H2) or TCP RST and let the CDN retry idempotent GETs on a healthy origin. Without that forced close, we’ve seen pools sit at 15-20 “zombie” connections for hours, blocking autoscaler scale-in.

If your drain never completes, don’t just extend the timeout — instrument it. Export active_connections, requests_in_flight, and drain_time_remaining from each origin and graph them during a deploy. When we debugged a stuck drain last month, the metric showed 120 connections stuck in CLOSE_WAIT because the app wasn’t closing DB-backed response streams. The fix wasn’t CDN-side at all, it was adding a 10-second server-side stream timeout. Also, be careful combining draining with aggressive hedging. If you’ve enabled Origin Request Hedging: Sending Parallel Requests to Cut Tail Latency to cut tail latency, a hedging race can accidentally send a duplicate request to the very origin you’re draining, effectively nullifying the drain. Most CDNs let you pin hedged retries to non-draining origins — turn that on, or you’ll see latency hedging create 503s during your most delicate window.

Finally, test your pool limits under drain load. When you drain one origin from a 4-origin pool with max_conns=1000 per origin, the other three suddenly take +33% traffic. If they’re already at 75% CPU, that burst will push p99 origin latency from 180ms to 600ms+ and trigger cascading timeouts. We run load tests at N-1 capacity with a sustained 2k RPS to confirm we can lose one origin without breaching our 500ms SLO. If you can’t, either over-provision by 25% or throttle drain concurrency to one origin at a time.

Frequently Asked Questions