CDN Configuration Drift Detection: Preventing Silent Edge Failures

The most dangerous CDN failures aren't the ones that crash loudly. They're the silent ones — a cache rule quietly modified through the dashboard, a WAF exception added during an incident and never removed, an origin timeout changed by someone who forgot to update the Terraform config. Configuration drift is the slow accumulation of undocumented changes that make your CDN's behavior diverge from what your team believes is deployed. When the next legitimate deployment runs, it either overwrites the undocumented changes (potentially breaking production) or fails with unexpected conflicts. This article covers how to build continuous drift detection for CDN infrastructure using GitOps state comparison, automated remediation, and prevention strategies.

How Drift Happens

Configuration drift has four primary sources in CDN infrastructure, each requiring different detection and prevention strategies.

Manual dashboard changes. Someone logs into the CDN provider's dashboard and modifies a setting. This is the most common source of drift. It happens during incidents when speed matters more than process, during debugging sessions when engineers test hypotheses by changing live settings, and when non-engineering teams (marketing, product) have dashboard access and change redirect rules or page rules without going through the standard deployment process.

Partial deployment failures. An infrastructure-as-code apply runs halfway and fails. Some resources are updated, others remain at the previous version. The Terraform state file may or may not reflect the partial changes, depending on where the failure occurred. This leaves the live CDN in an intermediate state that doesn't match either the old or new intended configuration.

Provider-side changes. CDN providers occasionally migrate settings, deprecate features, or apply automatic security updates. These changes happen without your involvement and aren't reflected in your infrastructure-as-code definitions. For example, a provider might automatically upgrade a minimum TLS version or modify the behavior of a deprecated feature flag.

Concurrent changes. Two engineers modify CDN configuration simultaneously through different mechanisms — one through Terraform, another through the API — and the changes conflict. The last write wins, but the "losing" change may still be in the Git repository as the expected state, creating drift.

Building a Drift Detector

A drift detector continuously compares the desired CDN state (from your Git repository or Terraform state) against the actual CDN state (queried from the provider's API). The comparison produces a diff — the specific settings that diverge — which drives remediation or alerting.

# drift_detector.py
import json
import subprocess
import hashlib
from datetime import datetime

class CDNDriftDetector:
    def __init__(self, terraform_dir, provider_api, zone_id):
        self.terraform_dir = terraform_dir
        self.provider_api = provider_api
        self.zone_id = zone_id

    def get_desired_state(self):
        """Extract desired state from Terraform"""
        result = subprocess.run(
            ["terraform", "show", "-json"],
            capture_output=True, text=True,
            cwd=self.terraform_dir
        )
        state = json.loads(result.stdout)
        return self._normalize_state(state)

    def get_actual_state(self):
        """Query live state from CDN API"""
        endpoints = [
            f"/zones/{self.zone_id}/pagerules",
            f"/zones/{self.zone_id}/firewall/rules",
            f"/zones/{self.zone_id}/settings",
            f"/zones/{self.zone_id}/dns_records",
            f"/zones/{self.zone_id}/cache/tiered_cache_smart_topology",
        ]

        state = {}
        for endpoint in endpoints:
            resp = self.api_get(endpoint)
            resource_type = endpoint.split("/")[-1]
            state[resource_type] = resp.get("result", [])

        return self._normalize_state(state)

    def detect_drift(self):
        desired = self.get_desired_state()
        actual = self.get_actual_state()

        diffs = []
        for resource_type in desired:
            if resource_type not in actual:
                diffs.append({
                    "type": "missing_in_live",
                    "resource": resource_type,
                    "expected": desired[resource_type]
                })
                continue

            desired_hash = self._hash(desired[resource_type])
            actual_hash = self._hash(actual[resource_type])

            if desired_hash != actual_hash:
                diffs.append({
                    "type": "modified",
                    "resource": resource_type,
                    "expected": desired[resource_type],
                    "actual": actual[resource_type],
                    "diff": self._compute_diff(
                        desired[resource_type],
                        actual[resource_type]
                    )
                })

        # Check for resources in live that aren't in desired state
        for resource_type in actual:
            if resource_type not in desired:
                diffs.append({
                    "type": "unexpected",
                    "resource": resource_type,
                    "actual": actual[resource_type]
                })

        return {
            "timestamp": datetime.utcnow().isoformat(),
            "drifted": len(diffs) > 0,
            "drift_count": len(diffs),
            "diffs": diffs
        }

    def _hash(self, obj):
        return hashlib.sha256(
            json.dumps(obj, sort_keys=True).encode()
        ).hexdigest()

    def _normalize_state(self, state):
        """Remove volatile fields that change without drift"""
        volatile_fields = ["modified_on", "created_on", "id"]
        return self._strip_fields(state, volatile_fields)

    def _strip_fields(self, obj, fields):
        if isinstance(obj, dict):
            return {
                k: self._strip_fields(v, fields)
                for k, v in obj.items()
                if k not in fields
            }
        if isinstance(obj, list):
            return [self._strip_fields(i, fields) for i in obj]
        return obj

Remediation Policies

Not all drift should be handled the same way. Define a remediation policy that classifies drift by risk level and assigns the appropriate response.

# remediation-policy.yaml
policies:
  auto_remediate:
    description: "Safe to automatically restore to desired state"
    resources:
      - cloudflare_page_rule
      - cloudflare_rate_limit
      - cloudflare_filter
      - dns_record (non-MX, non-NS)
      - cache_settings
    action: apply_desired_state
    notification: slack_info

  alert_and_block:
    description: "Security-sensitive — alert team and block deploys"
    resources:
      - cloudflare_firewall_rule
      - cloudflare_waf_rule
      - tls_settings
      - access_rule
      - ip_access_rule
    action: alert_security_team
    block_deploys: true
    notification: pagerduty

  report_only:
    description: "Informational — track but don't act"
    resources:
      - analytics_settings
      - development_mode
      - challenge_ttl
    action: log_to_audit
    notification: none

The policy distinguishes between drift that can be safely auto-fixed (cache rules, rate limits) and drift that requires human judgment (security rules, TLS settings). For security-sensitive drift, block all deployments until the drift is reviewed — deploying on top of unknown state is dangerous.

Terraform-Based Drift Detection

If you manage CDN infrastructure with Terraform, drift detection is built in. terraform plan compares the desired state in your configuration files against the live state tracked in the state file, and the live state is refreshed from the provider API at plan time.

# Automated drift check via Terraform
#!/bin/bash
set -euo pipefail

TERRAFORM_DIR="environments/production"
DRIFT_LOG="/var/log/cdn-drift/$(date +%Y%m%d_%H%M%S).json"

cd "$TERRAFORM_DIR"

# Refresh state from provider
terraform refresh -input=false 2>/dev/null

# Plan with detailed exit code
# Exit 0 = no changes, Exit 1 = error, Exit 2 = changes detected
terraform plan \
    -detailed-exitcode \
    -input=false \
    -json \
    -out=drift-check.plan \
    > "$DRIFT_LOG" 2>&1 || EXIT_CODE=$?

case ${EXIT_CODE:-0} in
    0)
        echo "No drift detected"
        ;;
    2)
        echo "DRIFT DETECTED — analyzing changes"

        # Extract changed resources from plan
        terraform show -json drift-check.plan | \
            python3 -c "
import sys, json
plan = json.load(sys.stdin)
changes = plan.get('resource_changes', [])
for change in changes:
    if change['change']['actions'] != ['no-op']:
        print(f\"  {change['address']}: {change['change']['actions']}\")
"
        # Notify team
        send_slack_alert "CDN drift detected" "$DRIFT_LOG"
        ;;
    *)
        echo "ERROR running drift check"
        send_pagerduty_alert "CDN drift check failed"
        ;;
esac

# Cleanup
rm -f drift-check.plan

Real-Time Drift Detection with Webhooks

Scheduled drift checks run every 5-15 minutes, meaning drift can exist for up to 15 minutes before detection. For critical configurations, implement real-time detection using the CDN provider's audit log webhooks.

# Real-time drift detection via audit log webhook
from flask import Flask, request

app = Flask(__name__)

@app.route("/webhook/cdn-audit", methods=["POST"])
def handle_audit_event():
    event = request.get_json()

    # Cloudflare audit log event structure
    action = event.get("action", {})
    actor = event.get("actor", {})
    resource = event.get("resource", {})

    # Check if change came from our automation
    if actor.get("email") == "[email protected]":
        # Expected change from our pipeline — no drift
        return {"status": "ok"}, 200

    # Manual change detected — potential drift
    drift_event = {
        "type": "manual_change",
        "actor": actor.get("email"),
        "action": action.get("type"),
        "resource_type": resource.get("type"),
        "resource_id": resource.get("id"),
        "timestamp": event.get("when"),
    }

    # Classify and respond
    if is_security_resource(resource.get("type")):
        alert_security_team(drift_event)
        block_deployments(reason="Manual security change detected")
    else:
        log_drift(drift_event)
        schedule_remediation(drift_event)

    return {"status": "processed"}, 200

Prevention Strategies

Detection is essential, but prevention is better. Reduce drift by making the correct change process easier than the bypass.

Lock down dashboard access. Most CDN drift comes from dashboard changes. Restrict write access to a small group of operators. For organizations with compliance requirements, use role-based access control to give most team members read-only dashboard access. Changes go through pull requests, not the dashboard.

Use API tokens with narrow scopes. If engineers need API access for debugging, issue read-only tokens. Write-capable tokens go only to the automation system. This is especially important for API access to security-sensitive features like WAF rules.

Emergency change process. The #1 reason people bypass GitOps is urgency — during an incident, the pull-request process feels too slow. Design an explicit emergency change process: make the dashboard change, document it in the incident channel, and file a follow-up PR within 24 hours to commit the change to Git. The drift detector will flag the change until the PR merges, which serves as a natural reminder to complete the process.

# Emergency change tracking
emergency_changes:
  - id: EC-2026-042
    date: "2026-10-01T03:45:00Z"
    actor: "[email protected]"
    change: "Disabled WAF rule 12345 blocking legitimate traffic"
    incident: "INC-789"
    followup_pr: "PR-456"  # Must be filed within 24h
    status: "committed"    # PR merged, drift resolved

Drift Metrics and Reporting

Track drift as a team metric. Measure drift frequency (how often drift is detected), drift duration (how long drift persists before detection), and drift source (who or what caused it). These metrics reveal process gaps and guide improvements.

Publish a weekly drift report showing all detected drift events, their sources, and remediation status. This creates visibility and accountability — teams that see their dashboard changes flagged in weekly reports tend to use the GitOps process more consistently over time.