CDN Drift Remediation Automation: Detecting and Fixing Configuration Divergence

I got paged at 2:17 AM because our image optimization rules had vanished from the Singapore edge. Someone had hot-fixed a cache TTL directly in the provider console during a Black Friday incident, and our Terraform state never knew it happened. Three weeks later a routine pipeline run re-applied the old configuration and wiped that manual fix, taking checkout down for 43 minutes. That's CDN drift in a nutshell, and if you aren't actively hunting for configuration divergence, you're just waiting for the next surprise.

The Console Isn't the Source of Truth (But Everyone Treats It Like One)

When we moved our edge configuration to Infrastructure as Code five years ago, I assumed we'd killed manual drift for good. Everything was in Git, peer-reviewed, and applied through CI. What I didn't account for was how often people bypass the pipeline when things are on fire. Support engineers, SREs, even our CDN vendor's professional services team — they all have console or API access, and when latency spikes or an origin goes dark, nobody's waiting for a pull request to merge. I've seen this on every platform from CloudFront to Fastly to Akamai: the live edge state slowly peels away from what's in version control until you don't have one source of truth, you have two.

The nasty part about CDN drift isn't that it happens, it's what drifts. It's never the harmless stuff. It's the cache-key normalization that got tweaked to fix a query-string bug, the security header that was added manually after a pen test and never committed, or the stale-while-revalidate directive someone disabled because they misunderstood a spec. Per the HTTP Working Group Specifications on caching, a single missing Cache-Control or Surrogate-Key header can completely change revalidation behavior at the edge. The web.dev Articles on caching best practices make the same point — inconsistent edge rules silently degrade performance long before they cause an outright outage. You won't see it in your IaC plan output because your IaC doesn't know it's wrong.

We started treating any console write as a drift event we had to detect, not a process failure to prevent with policy alone. Policies help, but detection is what saves you. Our first version was crude: a nightly job that pulled the live configuration via the provider API and diffed it against the rendered Terraform JSON. It was noisy and fragile, but it caught a critical divergence in our first week — a WAF rule that had been disabled manually and never re-enabled. That was enough to justify building something serious. If you're still managing CDN resources with point-and-click workflows, moving to CDN Resources in CloudFormation: AWS-Native Infrastructure as Code for Content Delivery or a similar declarative approach is step one, but it doesn't finish the job. You need automation that proves your live config still matches what you declared.

#!/usr/bin/env python3
# drift_check.py - Compare live CDN config vs. desired state
import hashlib
import json
import requests

def normalize_config(raw_config):
    """Strip ephemeral fields that always differ (etag, lastModified, ids)"""
    ignored_keys = {"etag", "lastModified", "version", "deployedAt"}
    return {k: v for k, v in raw_config.items() if k not in ignored_keys}

def fetch_live_config(api_url, token):
    resp = requests.get(f"{api_url}/v1/edge/config", headers={"Authorization": f"Bearer {token}"})
    resp.raise_for_status()
    return normalize_config(resp.json())

def load_desired_state(path="desired/edge.json"):
    with open(path) as f:
        return normalize_config(json.load(f))

def config_hash(config):
    canonical = json.dumps(config, sort_keys=True).encode()
    return hashlib.sha256(canonical).hexdigest()

live = fetch_live_config("https://api.cdn-provider.example", token="env.CDN_TOKEN")
desired = load_desired_state()

if config_hash(live) != config_hash(desired):
    print(f"DRIFT DETECTED: live={config_hash(live)} desired={config_hash(desired)}")
    # emit metric, open ticket, or trigger remediation workflow
else:
    print("No drift: live config matches desired state")

Building a Drift Detection Pipeline That Doesn't Cry Wolf

The biggest mistake I made with our second-gen detector was making it too sensitive. We hashed the entire edge service object and alerted on any difference. Within two days we had 200 alerts, all false positives. CDN providers inject fields you don't control — edge node counts, automatic certificate rotations, normalized header casing, timestamps. If you don't normalize those out, your drift remediation system becomes the thing everyone mutes. We've learned to do a semantic diff, not a byte-for-byte diff. We compare only the fields we own: origin definitions, cache behaviors, header manipulations, function associations, and TLS settings. Everything else gets excluded in a normalization layer before hashing.

Polling once a day isn't enough either. For our high-traffic properties, we run detection on two triggers: a scheduled reconcile every 15 minutes, and an event-driven check via webhook or EventBridge when the provider emits a configuration change event. Most modern CDNs will publish config change notifications — CloudFront uses CloudTrail + EventBridge, Fastly has event logs, Akamai has Property Manager events. Wiring those up cut our mean time to detect from hours to under three minutes. The pipeline itself hasn't changed much in two years: fetch live config, normalize, hash and deep-diff against Git's desired state, then publish a drift report to S3 and emit a Datadog metric. The fix for the noisy alerts was adding an allowlist of ignored paths and a severity classifier so we don't page for a non-material change like a provider-added comment field.

Tooling choice matters less than the pattern, but it shapes how you implement automated fixes. Our team standardized on TypeScript for the detection lambda because we were already using CDN Infrastructure with Pulumi: Type-Safe CDN Configuration in Real Programming Languages for edge logic, so we could reuse the same type definitions for desired vs. live state validation. Other teams I've worked with do the same job beautifully with CDN Automation with Ansible Playbooks: Configuration Management at Edge Scale for periodic state reconciliation, especially when they're managing hundreds of properties across multiple vendors. Don't try to build one massive script that handles detection and remediation together. Keep them separate — detection should be read-only, idempotent, and safe to run every few minutes. Remediation comes later and needs guardrails.

// drift-detector/handler.ts - Event-driven semantic diff
import { DiffEngine } from "./diff";
import { getLiveConfig } from "./providers/fastly";
import { getDesiredConfig } from "./state/git";

const IGNORED_PATHS = [
  "$.activeVersion.createdAt",
  "$.serviceId",
  "$.versions[*].updatedAt"
];

export const handler = async (event: any) => {
  const [live, desired] = await Promise.all([
    getLiveConfig(event.propertyId),
    getDesiredConfig(event.propertyId, event.commitSha)
  ]);

  const diff = new DiffEngine({ ignoredPaths: IGNORED_PATHS })
    .semanticDiff(live, desired);

  if (diff.hasChanges()) {
    await publishDriftReport({
      propertyId: event.propertyId,
      severity: diff.severity(), // low | medium | critical
      changes: diff.changes,
      automatedFixEligible: diff.isSafeToAutoRemediate()
    });
  }
  return { driftFound: diff.hasChanges(), changeCount: diff.changes.length };
};

Deciding What Gets an Automated Fix and What Needs a Human

Not all drift should be auto-remediated. That's a hard lesson. Early on we set our pipeline to automatically re-apply desired state on any divergence, and we reverted a legitimate emergency block that security had placed on a botnet's User-Agent. We undid their mitigation 90 seconds after they applied it. Now we classify drift into two buckets. Safe-to-fix drift is deterministic and low-risk to revert: TTLs, cache-key settings, header additions, compression flags. Anything involving WAF rules, origin failover logic, or TLS certificate bindings requires manual approval, even if we can detect it perfectly. That classifier runs on the diff output and decides whether to open a pull request, trigger a pipeline, or just alert.

Before we let any automated fix run, we measure. If you can't quantify divergence, you can't tune your thresholds without guessing. We track four metrics religiously, and they've completely changed how we prioritize edge work. MTTD (Mean Time to Detect) tells us if our event plumbing is working. Divergence rate tells us whether our process is leaking — are we seeing more manual edits over time? MTTR for auto-remediated drift shows the payoff of the automation, and failed deployments due to undetected drift is our canary for gaps in detection coverage. When we first started, we were detecting drift in about four hours on average and fixing it in another two by hand. After automation, those numbers look very different.

I tell teams to start with alerting and a dashboard before they enable a single automated fix. Let the detector run in shadow mode for two weeks. You'll learn which properties drift constantly and which fields are noisy. Once you've got confidence in your normalization and severity logic, enable auto-remediation for one safe category — I'd suggest cache headers — and gate it behind a Slack approval button for the first month. We've kept it that way for our critical commerce properties for over a year. It sounds slow, but it builds trust, and trust is what keeps the automated fix from being disabled after the first false positive.

Building the Remediation Runner That Actually Heals Drift

You've got detection — now comes the part where most teams stall. A remediation runner isn't just a script that slaps the Git state back onto the CDN. If it were, you'd be reverting legitimate emergency hotfixes and causing outages at 2 a.m. The runner we run in production is a reconciliation loop, not a blind overwrite. It pulls desired state from Git (main branch, tagged release), pulls actual state from the CDN control plane APIs every 90 seconds, diffs them with normalized JSON, and then decides: auto-merge, auto-remediate, or open a ticket.

The trick is idempotency and ordering. CDN configs aren't flat key-values. A single Akamai or Fastly service can have 400+ rules, and pushing them in the wrong order will invalidate cache or drop traffic. We don't shell out to Terraform on every drift event — that's too slow and too risky. Instead, our runner uses the provider's atomic API and only patches the divergent stanza. For origin configs and shielding rules, we auto-remediate within 45 seconds. For VCL/WAF logic, we never auto-remediate; we generate a pull request with the diff. That split cut our mean-time-to-remediation from 4.2 hours to under 3 minutes for safe drift, while our false-positive remediation rate dropped to 0.4%.

# drift_remediator.py - reconciliation loop (simplified)
import hashlib, time, requests

DESIRED_STATE_URL = "https://git.internal/api/v4/projects/cdn-config/repository/files/prod%2Fedge.json?ref=main"
CDN_API = "https://api.fastly.com/service/SERVICE_ID/version/active"

def normalize(config: dict) -> str:
    # strip defaults, sort rules, ignore API-added metadata
    config.pop("updated_at", None)
    config["rules"] = sorted(config.get("rules", []), key=lambda r: r["priority"])
    return hashlib.sha256(str(config).encode()).hexdigest()

def remediate(drift_path, desired_value):
    # patch only the divergent path, don't push full service version
    resp = requests.patch(f"{CDN_API}/patch", json={
        "path": drift_path,
        "value": desired_value,
        "comment": "auto-remediation: drift id 8f3a - reconciled to git@a1b2c3d"
    }, headers={"Fastly-Key": "TOKEN"})
    resp.raise_for_status()
    # verify + canary 5% of PoPs before full rollout
    time.sleep(10)
    assert normalize(requests.get(CDN_API).json()) == normalize(desired_value)

while True:
    desired = requests.get(DESIRED_STATE_URL).json()
    actual = requests.get(CDN_API).json()
    if normalize(desired) != normalize(actual):
        # classify drift - safe vs needs human
        if is_safe_to_autofix(desired, actual):
            remediate(find_drift_path(desired, actual), desired)
        else:
            create_jira_ticket(desired, actual)
    time.sleep(90)

Don't build this on top of ad-hoc templating. If your desired state is rendered at deploy time, store both the rendered output *and* the template variables in Git. We learned this the hard way when a Jinja2 default flipped `stale-while-revalidate` from 60s to 0s across 180 PoPs. Using consistent CDN Configuration Templating: Jinja2, HCL, and Go Templates for Edge Configs with strict linting and snapshot tests prevents your remediation from re-introducing drift. And if you're managing the underlying edge OS/packages alongside the CDN config, you'll want your remediation to respect the same source-of-truth boundaries — our edge nodes are converged via CDN Configuration with Chef: Recipe-Based Infrastructure for Content Delivery for cache nodes and CDN Puppet Management for Edge Nodes: Declarative Configuration at Scale for older bare-metal PoPs, so the CDN runner never touches system-level headers or kernel tuning.

Advanced Patterns: Circuit Breakers, Blast Radius Control, and When NOT to Auto-Fix

Automation without a circuit breaker is just a faster way to cause a global incident. Our biggest scare wasn't drift — it was the remediator itself. A bad merge to main propagated a broken `Cache-Key` rule that would have purged 72TB of hot cache. The runner caught the drift (good) and tried to fix 40 edge services in parallel (very bad). Now we enforce three gates: a global concurrency limit of 3 remediations per minute, a per-service canary that validates the fix on 2 PoPs in one region for 5 minutes before full rollout, and an automatic halt if error rate (5xx) spikes more than 0.15% or origin offload drops more than 8% in the 10 minutes post-remediation.

You'll also need to handle flapping and out-of-band changes correctly. Some drift is intentional and temporary — incident response tweaking TTLs from 3600s to 30s during an origin outage shouldn't be reverted mid-incident. We tag those changes with a `drift: acknowledgedUntil=2024-12-19T04:00Z` label in the CDN's metadata API. The runner ignores drift on that path until the window expires, then automatically re-converges. Without that, your on-call will simply disable the remediator, and then you've got no safety net at all. Track your `drift_acknowledged` vs `drift_remediated` metrics separately; if acknowledged drift exceeds 15% of total drift for more than 24 hours, you've got a process problem, not a tooling problem.

Finally, log everything as an auditable event, not just a Git commit. Every auto-fix should emit who (service account), what (JSON Patch RFC 6902 diff), why (drift ID + detector run ID), and the pre/post hash. We ship that to SIEM and keep it for 90 days. It's saved us twice during compliance audits where we had to prove that no human manually bypassed WAF rules — the remediation log showed the WAF drift was *rejected* for auto-fix and required two-person approval. If your pipeline can't prove it *didn't* auto-fix something dangerous, it won't pass a serious security review.

Frequently Asked Questions