Zero-Downtime Certificate Rotation: Rolling Updates Across CDN Edges

Back in 2019, we pushed a new wildcard cert to 2,400 edge nodes at once and watched our error rate spike to 8% in under 90 seconds. The cert itself was valid, the chain was correct, but half our fleet reloaded while handling 40k handshakes per second and dropped connections mid-flight. That incident taught me that certificate rotation isn't a file copy problem — it's a distributed systems problem where you can't afford to blink.

Why Hot-Swapping Certs Breaks at CDN Scale

On a single origin server, you can get away with overwriting a cert file and running nginx -s reload. At the edge, that approach falls apart fast. You've got thousands of PoPs, each with dozens of workers terminating TLS, each holding long-lived connections and cached sessions. If you replace the cert on disk and signal the process, you'll get a thundering herd of reloads where existing handshakes fail, session tickets become invalid, and clients see intermittent ERR_SSL_VERSION_OR_CIPHER_MISMATCH errors. I've seen teams blame the CA for what was really a deployment race condition.

The fix is to treat certificates as versioned, immutable bundles and load them atomically without dropping the listener socket. Most modern edge stacks — whether it's NGINX, Envoy, or a custom Rust terminator — support loading multiple certificates for the same SNI and switching via reference, not file mutation. You keep the old cert active while you validate the new one, then flip traffic. This pairs directly with OCSP Stapling for CDN: Eliminating Certificate Validation Delays, because if your new cert doesn't have a fresh stapled response, you'll just trade one outage for another. And don't forget that choosing between Wildcard vs SAN Certificates for CDN: Making the Right Choice changes how you bundle and distribute — a SAN with 80 names fails very differently than a wildcard when one edge lags behind.

# Envoy SDS-style atomic cert swap - don't overwrite, version and promote
# Directory layout on edge node:
# /etc/edge/certs/v2024-11-01-01/  (active)
# /etc/edge/certs/v2024-11-08-01/  (staged)

# 1. Stage new bundle with full chain + OCSP response
certctl stage --bundle /tmp/new-bundle.pem \
  --chain /tmp/chain.pem \
  --ocsp /tmp/ocsp.der \
  --version v2024-11-08-01

# 2. Validate without affecting live traffic
certctl verify --version v2024-11-08-01 \
  --sni www.example.com \
  --check-ocsp --check-sct --check-alpn

# 3. Atomic promote - control plane flips pointer, data plane reloads via SDS/xDS
certctl promote --version v2024-11-08-01 --strategy atomic-swap

# Old version kept for 24h for fast rollback
certctl list
# v2024-11-01-01  active (retiring in 23h)
# v2024-11-08-01  active (primary)

Orchestrating a True Rolling Update Across Regions

Once you can swap certs on a single node without drops, you need to roll that change across the fleet without taking the whole CDN down if something's wrong. We never push globally anymore. Our current pattern is canary -> regional batches -> global, with automatic health gates between each. I learned this the hard way after a bad intermediate forced us to roll back 1,800 nodes manually. Now our orchestrator won't proceed unless handshake success rate and p99 latency stay within bounds.

In practice, that means integrating cert deployment with your existing control plane rather than treating it as a separate cron job. We use a small coordinator service that watches the cert inventory and drives the rollout, very similar to how you'd automate renewal in Certificate Lifecycle Automation: From Issuance to Renewal at CDN Scale. It selects 2-3 PoPs per continent for the canary, waits 10 minutes, checks metrics, then expands to 25%, 50%, and 100% in waves. Each edge reports its active cert version via gossip, so we know exactly who hasn't converged. If you follow the guidance in HTTP Working Group Specifications for handling TLS session resumption and connection coalescing, you'll also want to verify that new certs don't break HTTP/2 and HTTP/3 handshakes during the transition — we've caught ALPN issues in canary that would have been brutal at full scale.

// Rolling cert deployment coordinator (Go pseudocode)
type RolloutPlan struct {
    Version string
    Batches []Batch // e.g. [{Canary: 3 PoPs}, {25%}, {50%}, {100%}]
    HealthGate HealthGate
}

func (c *Coordinator) Rollout(ctx context.Context, plan RolloutPlan) error {
    for i, batch := range plan.Batches {
        nodes := c.SelectNodes(batch)
        log.Printf("Batch %d/%d: deploying %s to %d nodes", i+1, len(plan.Batches), plan.Version, len(nodes))
        
        if err := c.DeployAndPromote(ctx, nodes, plan.Version); err != nil {
            return fmt.Errorf("deploy failed batch %d: %w", i, err)
        }
        // Wait for convergence and check SLOs
        if err := c.WaitConverged(ctx, nodes, plan.Version, 5*time.Minute); err != nil {
            c.Rollback(nodes) // Keep old version hot, just flip pointer back
            return err
        }
        if !c.CheckGate(ctx, plan.HealthGate, 10*time.Minute) {
            c.Rollback(nodes)
            return fmt.Errorf("health gate failed, rolled back batch %d", i)
        }
    }
    return nil
}

// Health gate example - queries Prometheus/ClickHouse for edge metrics
// handshake_success > 99.95%, p99 handshake < 80ms, no spike in 5xx

What We Measure Before We Call a Rotation Safe

You can't just check that the cert file exists and call it done. For zero downtime, we gate every batch on real client-facing signals, not just internal "reload succeeded" logs. Early on we relied on process exit codes and thought we were green, while users in Brazil were hitting expired OCSP staples because our cron to fetch them hadn't run on the newly promoted cert. Now we watch four things continuously for at least 10 minutes per batch, and we've got alerts that page if any of them drifts. The details in web.dev Articles on how browsers handle cert errors and interstitials helped us set realistic thresholds — users don't retry the way your synthetic monitor does.

The other lesson is to keep your old cert around and observable until you're sure. We retain both versions for 24-48 hours and expose the active version on a debug header (X-Edge-Cert-Version) that our RUM pipeline collects. That lets us correlate any TLS error with the exact cert version and PoP, instead of guessing if it's propagation lag. When we see even a 0.05% uptick in handshake failures in one region, we pause the rollout and investigate rather than hoping it'll self-heal. It almost never does.

Putting It Into Practice: Dual-Cert Hot Loading at the Edge

We never do an in-place overwrite of a cert file on a live edge. That's the fastest way to get inconsistent handshakes and a spike in ERR_SSL_VERSION_OR_CIPHER_MISMATCH errors. Instead, every edge node loads both the active and the incoming certificate into memory and negotiates based on what the client can handle. We've been running dual-cert slots since 2021, and it's the only approach that's given us truly hitless rotations at 400+ PoPs.

The mechanics are simple but strict. Your TLS terminator — whether that's NGINX, Envoy, or ATS — needs to watch a versioned cert directory, not a single file path. The orchestrator pushes cert-v2.pem and key-v2.pem alongside the active cert-v1, validates the chain and OCSP response locally, then flips an atomic symlink or fires an SDS update. Existing connections stay pinned to v1 via session ticket and connection state, while all new TLS handshakes negotiate with v2. We hold that dual-state window for at least 2x our session ticket lifetime — typically 24 to 48 hours — before we ever garbage-collect v1. It costs you an extra ~4-8MB RAM per worker, which is nothing compared to a 0.4% handshake failure rate.

Don't forget that changing certs can subtly alter your handshake fingerprint and record layer behavior. We've seen rotations where a switch from RSA 2048 to ECDSA P-256 shaved 35ms off the p95 handshake but tripped bot detection rules that keyed on JA3. If you're tuning performance right after a rotation, check your setup against TLS Fingerprinting Defense: Preventing Client Identification Through Handshake Analysis and TLS Record Size Optimization: Balancing Latency and Throughput at the Edge — we've had to adjust our dynamic record sizing from 4KB back to 16KB after a cert change to keep TTFB stable for long-tail video clients.

# Envoy SDS + control plane - hitless rotation pattern
# 1. Push new cert as a new SDS resource, don't mutate the old one

resources:
  - name: "edge_cert_v2"
    tls_certificate:
      certificate_chain: { filename: "/certs/cert-v2.pem" }
      private_key: { filename: "/certs/key-v2.pem" }
      ocsp_staple: { filename: "/certs/ocsp-v2.der" }

# 2. Control plane gradually shifts listener to v2 (weighted)
listeners:
  - name: "https_443"
    filter_chains:
      - transport_socket:
          name: envoy.transport_sockets.tls
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
            # Envoy will serve both during transition if both are in SDS cache
            common_tls_context:
              tls_certificate_sds_secret_configs:
                - name: "edge_cert_v1"
                - name: "edge_cert_v2"
              combined_validation_context: { ... }

# 3. Health check: verify v2 handshake before draining v1
# curl --resolve example.com:443:127.0.0.1 https://example.com -v --cert-status
# Expect: OCSP response: good, verify return code: 0, cert serial == v2

When Rotations Get Messy: Staggered Canaries and Fast Rollbacks

Even with dual-loading, you'll hit edge cases. The most common failure we see isn't expiration — it's intermediate chain mismatch. You push a shiny new cert signed by a G2 intermediate, but 3% of your edges still have the old CA bundle cached and serve an incomplete chain. Android 9 and older Java clients will hard-fail that handshake, and your global error rate won't catch it until it's wiped out traffic in São Paulo. That's why we never rotate globally. We do region-ringed canaries: 1 PoP in a single low-traffic region for 15 minutes, then 5% of traffic in that region, then the full region, then the next. If handshake failure rate ticks over 0.15% or OCSP stapling errors exceed 0.5% for 2 minutes, the orchestrator automatically rolls back the SDS pointer to v1. No human approval needed.

The other killer is long-lived connections — WebSockets, gRPC streams, and HTTP/3 QUIC sessions that can live for hours. They won't renegotiate TLS mid-session, so they’ll happily run on the old cert until they naturally drain. Don't force-close them. We've learned to let them drain for up to 72 hours while emitting a metric on tls_version_and_cert_serial. If you're managing pinning or private PKI, this drain window is where clients break hardest. We moved away from hard pinning years ago for this exact reason; if you haven't yet, read Beyond Certificate Pinning: Modern Alternatives for CDN Trust Verification — dynamic expect-CT plus short-lived certs with automated trust updates has saved us at least two sev-1s.

If you do get stuck, here’s our triage checklist: First, check ssl_stapling_file freshness — we pre-fetch OCSP 24 hours before rotation and require a staple younger than 6 hours before we promote v2. Second, diff the SAN and SCT list between v1 and v2; a missing SAN will cause 100% failure for that hostname but 0% globally, so your aggregate dashboards lie to you. Third, dump the actual wire handshake with openssl s_client -connect edge:443 -servername example.com -showcerts from inside the PoP, not from your laptop. We've chased “client bugs” for an hour that turned out to be a Puppet run reverting our cert permissions to 600 on 12% of nodes.

Frequently Asked Questions