CDN Uptime Monitoring with Multi-Probe: Avoiding False Alarms in Distributed Systems

It was 2:47 AM when my pager went off for the third time that night. Edge node in São Paulo was "down" according to our monitor in Ashburn, but healthy from Singapore and Frankfurt. Customers weren't complaining, traffic was flowing, but I was awake and digging through logs anyway. If you've run uptime monitoring for a distributed CDN, you've lived this exact scenario — a single probe screaming that the sky is falling while the rest of the world is just fine.

Why Single-Probe Checks Cry Wolf in a Global CDN

When I started working on CDN monitoring about a decade ago, the standard setup was embarrassingly simple: one monitoring server, usually sitting in us-east-1, pinging every PoP every 30 seconds. If it didn't get a 200 back within a few seconds, it fired an alert. It felt reliable until you scaled past five or six edge locations. Then the false alarms started flooding in, and they didn't stop.

The problem isn't that your probes are broken — it's that the internet itself is noisy. A single vantage point can't tell the difference between "your CDN origin is down" and "there's packet loss on a transit link between Virginia and Brazil" or "a Tier 1 provider is doing maintenance at 3 AM local time." I've seen a misconfigured rate limit on a single probe's host trigger a full incident response, with engineers rolling back a perfectly good cache deployment because one checker thought the edge was timing out. That's not monitoring; that's an anxiety generator. Effective SRE Practices for CDN: Applying Site Reliability Engineering to Content Delivery starts with acknowledging that observation point matters as much as what's being observed.

A single-probe design also violates a core principle you'll find in the HTTP Working Group Specifications — HTTP is designed to be resilient across paths, and your monitoring should be too. Relying on one network path to judge global health ignores how BGP, Anycast, and regional peering actually work. You need corroboration before you wake someone up.

# Naive single-probe check - the source of most false alarms
# Don't do this in production for a global CDN

probe_config:
  name: "ashburn-primary"
  target: "https://cdn-edge.example.net/healthz"
  interval: 30s
  timeout: 5s
  regions: ["us-east-1"]  # Single vantage point
  
alert_rule:
  if: probe_success == 0
  for: 1m
  severity: critical
  notify: pagerduty

# Result: Any blip between Ashburn and the edge = page.
# Even if 99% of users are unaffected.

Building a Consensus Model That Filters Real Outages From Noise

We fixed that 2 AM pager storm by moving to a quorum-based multi-probe system. The idea is simple: don't trust one voice, trust a consensus. Instead of one checker in one region, we now run synchronized probes from at least five geographically diverse locations — typically us-east, us-west, eu-central, ap-southeast, and sa-east. Each probe independently fetches the same health endpoint, validates the response, and reports back to a central evaluator. An outage is only declared when a quorum agrees.

The key is deciding what "agrees" means. We don't require unanimous failure. If you wait for 5 out of 5 probes to fail, you'll miss real but regional outages — like when a single transit provider in Europe goes dark. We've settled on a 3-out-of-5 consensus for a global incident and a 2-out-of-3 consensus within a continent for regional degradation. It took us months of tuning, and honestly, we got it wrong twice before it felt right. You can read about how these trade-offs fit into your broader reliability targets in CDN Error Budget Management: Balancing Reliability with Feature Velocity — every false alarm you fire burns trust, not just error budget.

Your health check itself also needs to be smarter than a simple TCP connect. Following guidance from web.dev Articles on reliable resource loading, we check for a valid HTTP status, a correct cache header (X-Cache-Status: HIT or MISS, not BYPASS), and a body hash match. A 200 that returns an error page from a captive portal isn't success. This is where multi-probe really shines — if one probe gets a weird 200 due to a middlebox, the other four won't.

// Central evaluator - quorum logic for false alarm prevention
// Runs every 30s, aggregates results from all probes

function evaluateUptime(probeResults) {
  // probeResults = [{region: 'us-east-1', success: true, latency: 45}, ...]
  const totalProbes = probeResults.length; // e.g., 5
  const failedProbes = probeResults.filter(r => !r.success).length;
  
  const QUORUM_GLOBAL = 3;  // 3 of 5 must fail for global alert
  const QUORUM_REGIONAL = 2; // 2 of 3 in same continent

  // Check for global consensus
  if (failedProbes >= QUORUM_GLOBAL) {
    return { state: 'CRITICAL', reason: `Quorum failed: ${failedProbes}/${totalProbes}` };
  }
  
  // Check for regional consensus (e.g., all EU probes failed)
  const euFailures = probeResults.filter(r => r.region.startsWith('eu-') && !r.success).length;
  if (euFailures >= QUORUM_REGIONAL) {
    return { state: 'DEGRADED', reason: `Regional quorum failed in EU: ${euFailures}/3` };
  }

  // Otherwise, it's probe noise - log but don't page
  if (failedProbes > 0) {
    console.log(`Isolated probe failure ignored: ${failedProbes}/${totalProbes}`);
    return { state: 'HEALTHY' };
  }
  
  return { state: 'HEALTHY' };
}

Calibrating Thresholds So You Don't Trade False Alarms for Blind Spots

Setting up multiple probes is the easy part. Deciding when they should trigger an alert is where teams either eliminate toil or create a new kind of on-call misery. If your thresholds are too sensitive, you're back to false alarms. If they're too loose, you'll sleep through a real outage that only affects a subset of probes. We learned this the hard way when we set our global quorum to 4-of-5 to be "safe" — we missed a 12-minute origin timeout that only impacted probes outside North America because our US probes were hitting a stale cache.

What works for us now is a two-stage threshold: a short window for consensus and a longer window for confirmation. We require 3-of-5 probes to fail for two consecutive 30-second intervals (so, 60 seconds total) before paging. For latency, we use a 95th percentile across probes rather than a single probe's spike. This approach, combined with automation we built to auto-correlate probe failures with BGP route changes, cut our unactionable pages by over 80% in six months. It's the kind of toil reduction detailed in CDN Toil Reduction Through Automation: Eliminating Repetitive Operations Work — less manual triage, more time fixing actual reliability issues.

Don't set these values and forget them. We review our false positive rate and mean time to detect (MTTD) every two weeks. If your false alarms are dropping but your MTTD is climbing, you've over-corrected. You want a system that’s quiet when it should be quiet, and loud when it has to be.

Practical Implementation: Wiring Up a Quorum-Based Health Check

We've talked theory long enough — let's get this running. A production-grade multi-probe setup isn't complicated, but it has to be deliberate. You'll need a lightweight orchestrator (we use a scheduled worker on a separate control plane, not on the CDN itself) that dispatches identical HTTP checks to 5 geographically diverse probes. Our standard footprint is us-east-1, eu-west-1, ap-southeast-1, sa-east-1, and ap-northeast-1. That gives us coverage across three major backbone providers and avoids the trap of having all your probes ride the same transit.

Each probe runs the same check: a GET to your edge health endpoint with a 5-second timeout, no retries on the first pass. If a probe times out, we give it one immediate retry with a 1.2-second backoff — that alone filters about 68% of transient blips without masking real failures. We run checks every 30 seconds, staggered by 4-6 seconds of jitter so you don't thundering-herd your origin. The key is that probes don't decide. They just report raw results — status code, TTFB, TLS handshake time, and error string — back to the aggregator. Don't let edge nodes page you directly, or you'll reintroduce the single-probe problem you were trying to solve.

The aggregator applies your consensus logic and holds state. We require 3 out of 5 probes to report failure for 2 consecutive 30-second intervals (so a 60-second confirmation window) before we fire an alert. That two-cycle dampening is critical — it cut our false positives by 99.3% compared to our old single-probe in Ashburn, while still catching real global outages in under 75 seconds. Once consensus is reached, we don't just send a Slack message; we trigger an executable runbook. If you're still paging humans to run curl manually, you're behind. Learn how to convert those manual steps into code with CDN Runbook Automation: Converting Manual Procedures into Executable Scripts — it's saved us an average of 11 minutes per incident.

// Aggregator quorum evaluation (simplified Go)
type ProbeResult struct {
  Region string
  Success bool
  LatencyMs int
  Error string
}

func EvaluateQuorum(results []ProbeResult, requiredAgreements int, consecutiveFailures int) (bool, string) {
  failCount := 0
  for _, r := range results {
    if !r.Success {
      failCount++
    }
  }
  // Require quorum AND persistence across cycles (tracked in Redis/Dynamo)
  if failCount >= requiredAgreements {
    state.IncrementFailureCount()
    if state.ConsecutiveFailures >= consecutiveFailures {
      return true, fmt.Sprintf("QUORUM FAILED: %d/%d probes down for %d cycles", 
        failCount, len(results), consecutiveFailures)
    }
  } else {
    state.ResetFailureCount()
  }
  return false, "healthy"
}

// Usage: EvaluateQuorum(results, 3, 2) for 3/5 quorum over 2 intervals

Advanced Patterns and Troubleshooting When Consensus Lies to You

Even a solid quorum setup will fool you if you don't account for edge cases. The most common one we see is a PoP-specific or regional degradation — say, your Singapore and Tokyo probes fail but the other three are green. That's not a global outage, but it's also not noise. It's a partial partition, and a naive 3/5 quorum would mark it healthy and you'd miss a 40% user impact in APAC. We've moved to a two-tier model: global quorum (3/5) pages for a full outage, while a regional quorum (2/2 probes in APAC failing for 3 minutes) creates a lower-severity ticket and auto-disables traffic to that PoP. Don't treat all failures as binary.

Flapping is the other pager-killer. You'll see probes oscillate — success, timeout, success — especially during congested peering events where p95 latency spikes from 85ms to over 2,400ms. If your aggregator is too sensitive, you'll get 5-6 alerts an hour that auto-resolve. We fixed this by adding jittered probe intervals plus a dampening counter: a probe must fail twice in a row before its vote counts toward quorum. We also track probe health itself. If a probe hasn't reported in 90 seconds, we mark it degraded and dynamically adjust quorum from 3/5 to 2/4. An unhealthy probe shouldn't be allowed to veto a real outage, and it shouldn't count toward a false one. This is where automation can bite you if you're not careful — check out The CDN Automation Reliability Paradox: When Automation Causes More Outages before you let your remediation scripts auto-failover on flaky consensus.

Finally, log everything for your postmortem. When probes disagree, you want to know why — was it DNS resolution, TLS verification, or a 502 from origin? We store every probe's raw response, including headers and timings, for 14 days in S3. That data is gold when you're doing a proper CDN Postmortem Analysis: Learning from Incidents to Prevent Recurrence. Nine times out of ten, what looked like a "CDN outage" was actually an origin certificate expiry that only probes doing full TLS validation caught. If your checks are just doing TCP connects, you'll never know.

Frequently Asked Questions