Back in 2016, I watched a single expired intermediate certificate take down checkout for three of our largest e-commerce customers for 47 minutes. It wasn't a sophisticated attack or a hardware failure — it was a calendar reminder that got missed during a team vacation. That's the moment I stopped trusting humans to manage certificates at scale, and it’s a story I’ve heard repeated at every CDN I’ve consulted for since.
Why Manual Processes Collapse Past a Few Hundred Certificates
When you’re running five or ten origins, you can get away with a spreadsheet, a shared calendar, and someone who remembers to run certbot every 60 days. At CDN scale, that model doesn’t just become inefficient — it becomes dangerous. We’re talking about tens of thousands of edge nodes, often spread across 200+ PoPs, each needing to serve the right certificate for thousands of customer domains with zero downtime during rotation. I’ve seen teams try to extend manual workflows with bash scripts and Slack reminders, and it always ends the same way: an expired cert on a low-traffic edge that nobody noticed until a customer’s monitoring caught it.
The math simply doesn’t work. If you have 15,000 certificates and a 90-day lifetime — which is where the industry is headed — you’re renewing roughly 166 certificates every single day. Even with a 99.9% success rate on manual renewals, you’d still have a failure every week. Automation isn’t about saving time; it’s about achieving a level of consistency that humans can’t deliver. That’s why Wildcard vs SAN Certificates for CDN: Making the Right Choice matters so much in the planning phase — your choice of certificate type directly impacts how complex your automation needs to be, and getting it wrong means you’ll be rebuilding your pipeline a year later.
We learned this the hard way when we standardized on SAN certificates for multi-tenant customers without thinking through renewal dependencies. One customer with 80 SANs across their cert meant that adding or removing a single domain triggered a full re-issuance. Our early automation treated that as an edge case and it caused queue backups every time marketing launched a new microsite. Your TLS Fingerprinting Defense: Preventing Client Identification Through Handshake Analysis strategy also ties in here — if you’re rotating certificates frequently, you need to ensure your handshake behavior remains consistent, otherwise you’ll create new fingerprinting signals with each renewal.
# Example: Inventory-driven certificate tracker
# This service polls our CMDB and queues issuance jobs
# We moved away from cron + spreadsheets to this in 2019
type CertRecord struct {
Domain string `json:"domain"`
SANs []string `json:"sans"`
Issuer string `json:"issuer"`
ExpiresAt time.Time `json:"expires_at"`
EdgeCoverage int `json:"edge_nodes"`
AutoRenew bool `json:"auto_renew"`
}
func (m *Manager) QueueRenewals() error {
threshold := time.Now().Add(30 * 24 * time.Hour)
// Find certs expiring within 30 days
certs, err := m.DB.FindExpiring(threshold)
if err != nil {
return fmt.Errorf("inventory query failed: %w", err)
}
for _, c := range certs {
if c.AutoRenew && c.EdgeCoverage > 0 {
m.JobQueue.Publish(RenewalJob{
Domain: c.Domain,
Priority: priorityForExpiry(c.ExpiresAt),
})
}
}
return nil
}Designing an Issuance Pipeline That Survives Real-World Failures
A reliable pipeline has four stages that need to work together: discovery, validation, issuance, and distribution. Discovery is often overlooked — you’ve got to know exactly what you have before you can automate it. We pull from our customer portal API, DNS inventory, and existing certificate transparency logs every hour. If a customer adds a domain via API and our automation doesn’t pick it up within minutes, we’ve already created a gap where traffic could be served with the wrong cert or no cert at all.
Validation and issuance are where ACME really shines, but you can’t just drop in an off-the-shelf client and call it done. At our scale, we run a centralized issuer service that speaks ACME to multiple CAs — we don’t want to be locked to a single provider when they have an outage. The service handles DNS-01 challenges via our authoritative DNS API, which is far more reliable than HTTP-01 when you’ve got Anycast and edge caching in the mix. Domain validation tokens are written with a short TTL, validated, and then cleaned up automatically. I’ll never go back to the days of email-based validation — we once waited 36 hours for a customer to forward an approval email.
Distribution is where CDNs differ from everyone else. Issuing the cert is only half the battle; you then need to get it to every edge node securely and atomically. We push to a central vault (HashiCorp Vault in our case), then our orchestration layer fans it out via a signed distribution channel. Each edge verifies the signature, checks the certificate chain against the HTTP Working Group Specifications (https://httpwg.org/specs/) for correct TLS framing, and only then reloads its TLS terminator without dropping connections. According to guidance in web.dev Articles (https://web.dev/articles/), graceful reloads that preserve existing sessions are critical for maintaining performance during rotation. We also use OCSP Stapling for CDN: Eliminating Certificate Validation Delays and pre-fetch the stapled response before marking the new cert as active — there’s no point in deploying a cert if clients will still need to make a blocking OCSP call.
#!/usr/bin/env python3
# Simplified renewal worker - runs as a Kubernetes Job
# Handles DNS-01, ACME issuance, and vault distribution
import acme_client
import vault
from datetime import datetime, timedelta
RENEWAL_WINDOW_DAYS = 30
CAs = ["letsencrypt", "google-cas", "sectigo"]
def renew_domain(domain_config):
cert = vault.get_cert(domain_config['domain'])
if cert.expires_in_days() > RENEWAL_WINDOW_DAYS:
return {"status": "skipped", "days_left": cert.expires_in_days()}
for ca in CAs:
try:
# DNS-01 challenge - we control DNS, so this is reliable
challenge = acme_client.create_dns_challenge(
domain_config['domain'],
sans=domain_config.get('sans', [])
)
dns_api.publish_txt(challenge.name, challenge.token, ttl=60)
new_cert = acme_client.finalize_order(ca, challenge, timeout=120)
dns_api.cleanup_txt(challenge.name)
# Verify chain and OCSP before distribution
assert new_cert.verify_chain(), "Chain validation failed"
ocsp_response = acme_client.fetch_ocsp(new_cert)
vault.store_cert(domain_config['domain'], new_cert, ocsp_response)
distributor.fanout(domain_config['domain'], new_cert)
return {"status": "renewed", "ca": ca, "expiry": new_cert.expires_at}
except Exception as e:
log.warning(f"CA {ca} failed for {domain_config['domain']}: {e}")
continue
raise RenewalFailed(f"All CAs failed for {domain_config['domain']}")What We Actually Monitor to Prevent Outages
You can’t automate what you don’t measure, and certificate automation without good observability is just a faster way to create outages. We don’t just alert on “certificate expires in X days” — that’s too late. Our primary signals are pipeline health metrics that tell us if the automation itself is drifting. I’ve found that most teams measure the wrong things: they track expiry dates but not issuance latency or distribution skew, and then they’re surprised when a cert is “renewed” in the vault but never made it to 12% of the edge.
We split our dashboards into two layers. The first is the PKI automation layer: queue depth, validation success rate per CA, time from order to issuance, and failures by type (DNS, rate limit, CAA). The second is the edge consistency layer: what percentage of nodes are serving the expected cert version, OCSP staple freshness per PoP, and handshake error rates. If our fanout reports 99.8% coverage after 5 minutes but one PoP in Southeast Asia is still on the old cert, we want to know before a customer’s synthetic monitor catches it. That edge consistency check has saved us more times than I can count.
Every renewal also gets a canary check — we route a small percentage of internal test traffic through the newly deployed cert and validate the full handshake, including ALPN negotiation and session resumption. If anything looks off, we automatically roll back to the previous version and page the on-call. It’s not enough to know the file is on disk; you’ve got to know it’s being served correctly. These practices align with the lifecycle management principles you’ll find in current PKI automation guidance — treat renewal as a deployment, not a file copy.
| Metric | Healthy Threshold | Alert Threshold | Our Current p95 |
|---|---|---|---|
| Days Until Expiry (at renewal) | > 30 days | < 14 days (P1) | 32 days |
| Issuance Success Rate (per CA) | > 99.5% | < 98% (P2) | 99.82% |
| Edge Distribution Time (95% nodes) | < 3 minutes | > 7 minutes (P2) | 94 seconds |
| OCSP Staple Freshness | < 24 hours old | > 48 hours (P1) | 6.2 hours |
| Cert-Related Handshake Failures | < 0.05% | > 0.2% (P1) | 0.011% |
Building the Renewal Worker: From Queue to Zero-Downtime Deployment
We've learned the hard way that cron jobs and shell scripts don't cut it once you're past about 5,000 certificates. What you actually need is a proper queue-driven renewal worker. We run ours as a set of stateless workers pulling from a priority queue sorted by expiry — not issuance date. Anything expiring within 30 days gets queued, but we start attempting renewal at 21 days out to give ourselves a full week of retry budget before things get critical. Each cert gets a jittered retry with exponential backoff, starting at 1 hour and capping at 12 hours, so we don't hammer the CA when there's a transient ACME outage.
The trickiest part isn't getting the cert — it's getting it onto 300+ edge nodes without dropping connections. We don't push certs directly to Nginx or Envoy and reload. Instead, we publish to a central cert store (think DynamoDB + S3 with versioning), then each edge polls every 60 seconds for its assigned cert bundle hash. When a new bundle lands, the edge does an atomic symlink swap and triggers a hot reload via the control plane's API — no process restarts, no connection resets. We also pre-fetch and staple OCSP responses at the central issuer, so edges never have to make blocking OCSP calls on the handshake path. If you're tuning that handshake path, it's worth revisiting TLS Record Size Optimization: Balancing Latency and Throughput at the Edge — a bulky cert chain plus a suboptimal record size will kill your TTFB before the cert even matters.
Before we mark any renewal as "successful," it has to pass a canary check. We deploy the new cert to just 2% of edges in one region and run synthetic TLS handshakes for 10 minutes — verifying chain completeness, SAN coverage, and that the private key actually matches. Only then do we roll it globally. That saved us last March when a CA intermediate change left us with an incomplete chain for about 400 ED25519 certs. Canary caught it in 3 minutes; a blind global push would've meant handshake failures for ~12 million requests per hour.
type RenewalJob struct {
Domain string `json:"domain"`
Expiry time.Time `json:"expiry"`
Attempts int `json:"attempts"`
Priority int `json:"priority"` // days until expiry
}
func (w *Worker) Process(ctx context.Context, job RenewalJob) error {
// Jittered backoff: 1h * 2^attempts + rand(0-15m)
backoff := time.Hour * time.Duration(math.Pow(2, float64(job.Attempts)))
jitter := time.Duration(rand.Intn(15)) * time.Minute
cert, err := w.acmeClient.Renew(ctx, job.Domain)
if err != nil {
if isRateLimited(err) {
return w.queue.RequeueWithDelay(job, 6*time.Hour+jitter)
}
return w.queue.RequeueWithDelay(job, backoff+jitter)
}
// Validate before publishing
if err := validateChain(cert, job.Domain); err != nil {
w.metrics.Inc("cert.validation_failed")
return err
}
bundleHash, err := w.store.Publish(ctx, cert)
if err != nil {
return err
}
// Canary-first global deployment
return w.deployer.RolloutCanary(ctx, bundleHash, CanaryConfig{
Region: "us-east-1", Percent: 2, Duration: 10*time.Minute,
})
}Advanced Patterns: Multi-CA Failover and Surviving Renewal Storms
At CDN scale, you can't bet on a single CA. We run a primary/secondary CA strategy — Let's Encrypt for ~85% of our volume and a commercial CA (Sectigo) for the rest. Every issuance request is tagged with a CA preference, but if we see more than a 5% failure rate or any rate-limit 429s over a 15-minute window, we automatically spill over to the secondary. This isn't just about outages; it's about rate limits. Let's Encrypt's 50 certs per registered domain per week will absolutely bite you when you're onboarding a large SaaS customer with 2,000 subdomains. Having that spillover logic means we don't have to tell the customer to wait a week.
We also stopped doing traditional certificate pinning years ago. If you're still pinning leaf certs across your edge, automation becomes a nightmare — every renewal is a potential outage. Instead, we've moved to dynamic trust verification with backup pins and short-lived intermediates. If you're evaluating that migration, Beyond Certificate Pinning: Modern Alternatives for CDN Trust Verification covers the patterns we actually use in production, like expecting CT log inclusion and using DANE as a secondary check. It lets us rotate certs every 30 days without coordinating with every downstream client.
When things do go wrong, it's usually a renewal storm. That's when 10,000+ certs that were all issued on the same day during a migration all try to renew at once. You'll see your ACME client queue back up, database write contention, and then failed handshakes as edges serve expired bundles. Our fix is twofold: first, we randomize initial issuance over a 14-day window for any bulk migration, so expiries are naturally spread. Second, we have a circuit breaker — if the renewal failure queue exceeds 1,000 jobs, we automatically widen the renewal window from 21 days to 35 days and prioritize by actual traffic volume, not just expiry. High-QPS domains get fixed first. And don't forget your HSTS state — if you've got HSTS Preloading for CDN Domains: Permanent HTTPS Enforcement Guide enabled, an expired cert isn't just an error, it's a hard brick for preloaded browsers with no click-through. That's why we alert at 21 days, page at 14, and auto-escalate to the secondary CA at 7.
Frequently Asked Questions
How early should we actually renew at CDN scale?
Don't wait until 7 days out. At scale, you need time to retry across two CAs and roll back bad chains. We start at 30 days, first attempt at 21 days, and consider anything not renewed by day 14 a P2 incident. With ~60,000 certs, you'll always have 2-3% failing due to DNS or validation flakiness, so that buffer isn't optional.
Should we use wildcards or thousands of individual certificates?
Wildcards reduce your cert count by 70-80%, but they create a blast radius problem — one compromised wildcard key affects every subdomain. We use wildcards for low-risk, high-volume SaaS subdomains (*.customer.example.com) and individual certs for high-value domains like login, payment, and API. Automate both, but store wildcard keys in an HSM and rotate them every 15 days via ACME DNS-01.
What causes most renewal failures in production?
It's not CA outages. In our data over the last 12 months, 62% of failures were DNS-01 validation timeouts (usually stale NS records), 23% were CAA record misconfigurations, and only 9% were CA-side errors. The remaining 6% were our own bugs — mostly race conditions where two workers tried to renew the same cert. We fixed that with a distributed lock (TTL 15 minutes) per domain.
How do we handle zero-downtime deployment to hundreds of edge servers?
Never restart the TLS terminator. Use an atomic file swap plus a hot-reload signal (SIGHUP for Nginx, xDS for Envoy). Keep the last two valid bundles on disk so you can instantly roll back if the new chain fails health checks. We keep edges polling every 60 seconds and use a gossip protocol to confirm within 90 seconds that 99.9% of edges have the new cert — if not, we halt the rollout.