Health Checks and Failover in CDN: Ensuring 99.99% Uptime

Health checks are automated probes that continuously verify whether CDN servers, origins, and network paths are functioning correctly. Failover is the process of redirecting traffic away from unhealthy components to healthy alternatives. Together, these mechanisms form the backbone of CDN reliability, enabling the 99.99% uptime guarantees that modern applications demand. Without robust health checking and failover, a single server failure could take down content delivery for millions of users. For a comprehensive overview, see our guide on CDN Architecture: Edge, Origin, and Everything Between — FlyFish NWA.

This guide covers the types of health checks used in CDN infrastructure, how failover strategies work at each network layer, the math behind uptime SLAs, and how to prevent cascading failures that can turn a minor incident into a major outage.

What Are Active Health Checks and How Do They Work?

Active health checks send synthetic probe requests to servers at regular intervals to verify they are operational. The health checker initiates a connection to each monitored server, sends a predefined request, evaluates the response against expected criteria, and records the result. A server that fails a configured number of consecutive checks (the failure threshold) is marked unhealthy and removed from the active pool.

Active health checks operate at three protocol levels, each providing different depth of verification:

TCP Health Checks

TCP checks verify that a server accepts connections on a specific port. The checker initiates a TCP three-way handshake and considers the server healthy if the connection completes successfully. TCP checks are lightweight and verify that the server process is running and accepting connections, but they cannot detect application-level failures such as a web server returning errors or serving stale content.

# TCP health check configuration example
health_check {
  type: tcp
  port: 443
  interval: 10s
  timeout: 5s
  unhealthy_threshold: 3
  healthy_threshold: 2
}

HTTP and HTTPS Health Checks

HTTP checks send a full HTTP request (typically GET) to a specific endpoint and validate the response. A comprehensive HTTP health check verifies the status code (expecting 200), checks response headers, and optionally validates that the response body contains expected content. HTTPS checks add TLS certificate validation, ensuring the server presents a valid, unexpired certificate.

# HTTP health check configuration example
health_check {
  type: https
  path: /health
  host: cdn-edge.example.com
  interval: 15s
  timeout: 5s
  expected_status: 200
  expected_body_contains: "status: ok"
  unhealthy_threshold: 3
  healthy_threshold: 2
}

A well-designed health endpoint checks more than just the application process. It should verify that the server can reach its cache storage, that disk space is adequate, that the TLS certificate is valid, and that the server's upstream connections (to L2 caches or origin) are functional. This deep health check prevents scenarios where a server is technically running but unable to serve content correctly.

Custom Application Health Checks

Some CDN deployments implement custom health checks that go beyond HTTP verification. These might include verifying that the cache hit ratio is above a minimum threshold, that origin connectivity is functional, or that specific content is being served correctly. Custom checks typically run as scripts that the health check system executes periodically.

What Are Passive Health Checks and How Do They Complement Active Checks?

Passive health checks (also called real-time or inline monitoring) analyze actual user traffic rather than synthetic probes. The load balancer tracks the success and failure rates of real requests being routed to each server. When a server's error rate exceeds a threshold (for example, 50% of requests returning 5xx errors within a 30-second window), it is marked unhealthy.

Passive checks offer two key advantages over active checks. First, they detect failures faster during high-traffic periods because every user request acts as a health probe. A server failing 100 requests per second will be detected within seconds, whereas an active check with a 15-second interval might not fire for another 10 seconds. Second, passive checks catch failures that synthetic probes might miss, such as performance degradation under load or errors that only occur with specific request patterns.

The limitation of passive checks is that they cannot detect failures on servers that receive no traffic. A newly added server or one that has been drained of traffic will never fail a passive check because there are no requests to monitor. This is why production CDN architectures combine both active and passive health checking for comprehensive coverage.

How Should You Configure Check Intervals and Thresholds?

Health check intervals and thresholds directly control the tradeoff between detection speed and stability. The key parameters are:

A common production configuration uses a 10-second interval, 5-second timeout, 3 consecutive failures to mark unhealthy, and 2 consecutive successes to mark healthy again. This detects a hard failure within 30 seconds (three checks at 10-second intervals) while tolerating occasional network blips that might cause a single check to timeout.

For critical paths where faster detection is required, combine aggressive active checks (5-second intervals, 2 failure threshold) with passive monitoring. The passive checks provide sub-second detection during traffic, while the active checks catch failures during low-traffic periods.

What Failover Strategies Do CDNs Use?

When health checks detect a failure, the failover strategy determines how traffic is redirected to healthy alternatives. CDNs employ three primary failover mechanisms, each operating at a different network layer.

DNS-Based Failover

DNS-based failover removes the IP address of a failed server or data center from DNS responses. Subsequent DNS queries return only healthy endpoints. The limitation is propagation delay: clients that have cached the old DNS response continue sending traffic to the failed server until the DNS TTL expires. With a 60-second TTL, full failover can take up to 60 seconds. Some clients and resolvers ignore TTL values entirely, extending the failover window further.

To minimize DNS failover delay, CDNs use short TTLs (30-60 seconds) on their A and AAAA records. This increases DNS query volume but ensures faster traffic redirection during incidents. The tradeoff between DNS performance and failover speed is a fundamental tension in CDN DNS architecture.

BGP-Based Failover

In anycast deployments, BGP-based failover happens automatically when a failed location withdraws its BGP route advertisements. Traffic immediately shifts to the next nearest location that is still announcing the route. BGP convergence typically takes 5-30 seconds, making this significantly faster than DNS failover. No client-side caching can delay the process because the routing change happens within the internet's core infrastructure.

Application-Layer Failover

Load balancers perform application-layer failover by removing unhealthy servers from their active pool. This is the fastest failover mechanism: once a server fails health checks, the load balancer stops routing new connections to it immediately. Existing connections may be allowed to drain gracefully or may be terminated and retried on a healthy server, depending on configuration.

The most resilient CDN architectures combine all three failover mechanisms in layers. Application-layer failover handles individual server failures within a PoP (seconds). BGP failover handles PoP-level outages (5-30 seconds). DNS failover handles regional or provider-level outages (30-300 seconds).

How Do You Prevent Cascading Failures?

Cascading failures represent the most dangerous failure mode in CDN infrastructure. They occur when the failure of one component causes remaining components to become overloaded, which causes them to fail, creating a domino effect that can take down an entire region or the whole network.

A classic cascading failure scenario: a CDN has three data centers, each handling 40% capacity (with 20% headroom). When one data center fails, its traffic splits between the remaining two, pushing each to 60% capacity. This exceeds their ability to handle the additional connections and cache misses from the cold cache, causing response times to spike. As response times increase, client retry storms amplify the load further, causing the second data center to fail, and then the third.

Prevention strategies include:

• Capacity headroom: Maintain at least 50% headroom so that losing one data center does not overload the survivors. The N+1 redundancy principle means you provision for N times your peak traffic plus one additional unit of capacity.
• Circuit breakers: When a server or data center starts returning errors above a threshold, circuit breakers stop sending it traffic entirely rather than continuing to probe it with real user requests. This prevents a slow or erroring server from degrading the experience for every user whose request is routed there.
• Gradual traffic shifting: Rather than instantly redirecting 100% of a failed server's traffic to remaining servers, shift traffic gradually over 30-60 seconds. This gives receiving servers time to warm their caches with the new request patterns.
• Rate limiting on failover: Cap the additional traffic that healthy servers accept during failover to prevent overload. Excess traffic receives a synthetic error page from the load balancer rather than overwhelming backend servers.
• Origin protection: When edge cache failures cause a spike in origin requests, origin protection mechanisms such as request coalescing and origin shields prevent the origin from being overwhelmed by cache stampedes.

How Is CDN Uptime Calculated and What Do SLA Numbers Mean?

CDN Service Level Agreements (SLAs) express availability as a percentage of total time. The calculation is straightforward:

Uptime % = ((Total Minutes - Downtime Minutes) / Total Minutes) x 100

The practical impact of each "nine" of availability is significant:

Most major CDN providers offer 99.9% to 99.99% SLAs. Achieving 99.99% requires that all failover mechanisms combined limit total downtime to under 4.4 minutes per month. This means failures must be detected within seconds and traffic rerouted before users are significantly impacted.

It is important to understand how CDN providers define "downtime" in their SLAs. Some measure it from the CDN's perspective (edge server availability), while others measure it from the user's perspective (successful content delivery). Origin server outages may or may not count, depending on the SLA terms. Always read the SLA's definition of "unavailability" before relying on the percentage number.

How Do You Integrate Health Monitoring with Observability?

Health checks provide binary status (healthy or unhealthy), but comprehensive CDN monitoring requires richer observability data. Integrating health check results with metrics, logs, and traces creates a complete picture of CDN health.

Key metrics to track alongside health check status include response latency percentiles (p50, p95, p99), error rates by status code, cache hit ratio per server, origin fetch latency, and connection pool utilization. When these performance metrics degrade before health checks fail, they serve as early warning signals that allow operators to intervene before users are impacted.

Alerting should be layered. The first layer alerts on degraded metrics (elevated latency, declining cache hit ratio). The second layer alerts on health check failures. The third layer alerts on failover events. This graduated approach ensures that operators are notified early enough to act before automated failover fires and provides context about why the failover occurred.

The combination of proactive health checking, multi-layer failover strategies, cascading failure prevention, and integrated observability is what separates CDNs that deliver genuine four-nines availability from those that merely promise it. Each component is individually straightforward, but building a system where they all work together reliably under stress requires careful engineering and continuous testing through chaos engineering practices and planned failover drills.