CDN Puppet Management for Edge Nodes: Declarative Configuration at Scale

Back in 2019 I watched a routine Nginx cache key change turn into a 14-hour incident because it was pushed with a bash loop over 600 edge nodes in three regions. Half the nodes got the new config, half timed out, and three had a typo that wasn't caught until customers in São Paulo started seeing 50% cache misses. That's the moment my team stopped treating edge configuration like a scripting problem and started treating it like a state problem.

Why Imperative Scripts Fall Apart on a Large Edge Fleet

When you're managing 50 nodes, SSH loops and Ansible ad-hoc commands feel fast enough. When you're managing 5,000 nodes across 40 PoPs, that approach creates snowflakes you can't track. Imperative tools tell a server *how* to change, but they don't guarantee *what* state it ends up in if the run fails halfway. I’ve seen nodes where `apt-get` hung, leaving half-installed Varnish versions serving traffic for days because no subsequent check enforced the desired state. Declarative configuration flips that model. You define the final state you want, and the agent figures out how to get there and continuously corrects drift.

That's where Puppet earns its keep for CDN infrastructure. Unlike push-based systems that fire and forget, Puppet's pull-based agent runs every 30 minutes, compares the node's actual state to the compiled catalog from the Puppet master, and fixes anything that wandered. For edge nodes that handle HTTP Working Group Specifications caching semantics and strict header handling, that consistency is non-negotiable. If someone manually tweaks a `Cache-Control` override or disables a TLS cipher on one node, Puppet reverts it on the next run without you having to detect it first. Compared to alternatives like CDN Automation with Ansible Playbooks: Configuration Management at Edge Scale, which I've used for one-off orchestrations, Puppet's strength isn't speed of execution — it's the guarantee that your 5,000th node looks exactly like your first one.

We learned this the hard way during a POP expansion in Singapore. Our old bash-based provisioning took about 47 minutes per node and required a senior engineer to babysit it. The Puppet-based image we replaced it with converges a bare-metal edge node in under 8 minutes with zero manual intervention. It doesn't just install packages; it enforces file contents, service states, kernel tuning, and cron jobs as a single atomic declaration. If any resource fails, the whole run is reported as failed, and we don't mark that node as healthy in our load balancer.

# profiles/manifests/cdn/edge_baseline.pp
class profiles::cdn::edge_baseline (
  String $cdn_env = 'production',
  Array[String] $ntp_servers = ['0.pool.ntp.org', '1.pool.ntp.org'],
) {
  # Enforce baseline packages and kernel tuning for edge caching
  class { 'varnish':
    version => '7.3.0',
    vcl_template => "profiles/cdn/varnish-${cdn_env}.vcl.erb",
    malloc_size  => '80%',
  }

  sysctl { 'net.core.somaxconn':
    ensure => present,
    value  => '65535',
  }

  sysctl { 'net.ipv4.tcp_tw_reuse':
    ensure => present,
    value  => '1',
  }

  service { 'varnish':
    ensure  => running,
    enable  => true,
    require => Class['varnish'],
  }

  # Ensure no manual drift - puppet will replace unauthorized changes
  file { '/etc/varnish/default.vcl':
    ensure  => file,
    owner   => 'root',
    group   => 'root',
    mode    => '0644',
    content => template($varnish::vcl_template),
    notify  => Service['varnish'],
  }
}

Writing Puppet Manifests That Survive Real Edge Conditions

The biggest mistake I see teams make is writing manifests that work in the lab but not on a congested edge node with a full cache and 80k requests per second. You can't just restart Nginx or Varnish whenever a config changes — you'll drop connections. Good edge manifests are idempotent *and* service-aware. We use Puppet's `notify` and `refreshonly` patterns heavily so services only reload when the underlying file actually changes, and we use staged reloads with health checks. I also separate concerns using the roles/profiles pattern religiously: a `role::cdn::edge_cache` might include `profiles::cdn::edge_baseline`, `profiles::cdn::tls_termination`, and `profiles::cdn::monitoring`, but each profile is independently testable.

Another lesson from production: don't hardcode PoP-specific data in your manifests. We did that early on and ended up with a manifest full of `if $hostname =~ /^fra/` conditionals that became unmaintainable. Now we push all variance into Hiera. Cache sizes, upstream origins, and even `stale-while-revalidate` values from web.dev Articles research come from Hiera lookups keyed by datacenter, hardware tier, and role. That way the same manifest can provision a 256GB RAM node in Ashburn and a 64GB node in Warsaw without branching logic. It’s also what makes testing sane. We run `puppet catalog compile` in CI for every combination of facts before anything hits canary nodes. If you’re evaluating other declarative stacks, CDN Infrastructure with Pulumi: Type-Safe CDN Configuration in Real Programming Languages takes a similar approach but with real programming languages, which some teams prefer for complex logic.

Finally, handle secrets and certificates correctly. Edge nodes terminate TLS, so they need private keys, but you shouldn't be dropping them as plain files via Puppet without protection. We use Hiera eyaml with per-PoP keys and the `puppetlabs/stdlib` `deferred` functions so keys are never logged in reports. And always set `show_diff => false` for those files. I once debugged an issue where a misconfigured report processor was logging full VCL files with backend tokens into Splunk — not a great day.

# roles/manifests/cdn/edge_cache.pp
class roles::cdn::edge_cache {
  include profiles::base::hardening
  include profiles::cdn::edge_baseline
  include profiles::cdn::tls_termination
  include profiles::cdn::nginx_cache

  # Hiera-driven variance - no if/else on hostname
  $cache_size  = lookup('profiles::cdn::nginx_cache::cache_size')
  $origin_map  = lookup('profiles::cdn::nginx_cache::origins')
  $hsts_max_age = lookup('profiles::cdn::tls_termination::hsts_max_age', Integer, 'first', 31536000)

  class { 'profiles::cdn::monitoring':
    graphite_endpoint => lookup('graphite::endpoint'),
    service_checks    => ['varnish', 'nginx', 'haproxy'],
  }

  # Graceful reload only on change, with health check
  exec { 'nginx-config-test-and-reload':
    command     => '/usr/sbin/nginx -t && /bin/systemctl reload nginx && /usr/local/bin/check_nginx_health.sh',
    refreshonly => true,
    subscribe   => File['/etc/nginx/nginx.conf'],
    timeout     => 60,
  }
}

# Hiera example: data/nodes/fra-edge-042.yaml
# profiles::cdn::nginx_cache::cache_size: '120g'
# profiles::cdn::nginx_cache::origins:
#   api: 'origin-api.example.com'
#   static: 'origin-static.example.com'

Classifying Thousands of Nodes Without Losing Your Mind

Puppet's real power at CDN scale isn't the manifest language itself, it's classification and reporting. With edge node management, you can't afford to manually assign classes to each host. We use a Puppet External Node Classifier (ENC) that queries our inventory service (NetBox) every time a node checks in. When a new server boots in our Osaka PoP, it registers its facts — memory, disks, region, hardware generation — and the ENC returns its role, environment, and Hiera hierarchy automatically. No one edits `site.pp` anymore. This is fundamentally different from a cloud-native approach like CDN Resources in CloudFormation: AWS-Native Infrastructure as Code for Content Delivery, and it’s why we keep Puppet for bare-metal edge while using CloudFormation for our control plane in AWS.

The second half of that equation is visibility. PuppetDB stores every fact, catalog, and report, and we feed that into Grafana and PagerDuty. I don't just want to know if Puppet runs failed; I want to know if 5% of nodes in a PoP are correcting the same file on every run — that's a sign of a bad package or someone running a manual cron job that fights Puppet. We've built alerts on metrics like `catalog compilation time`, `agent run duration`, and `resources corrected per run`. When we migrated from 1,200 to 4,800 nodes last year, our average catalog compilation time crept from 1.2 seconds to over 4 seconds, which was throttling our Puppet masters. We fixed it by moving to Puppet Server with JRuby and enabling static catalogs, but we only caught it because we were tracking the right numbers.

If you aren't measuring convergence, you're just hoping your CDN is consistent. Hope doesn't keep cache hit ratios stable at 92% during peak.

Deploying Puppet to Production: Canaries, Ramps, and Safe Rollouts

Getting a manifest to work on your laptop is one thing. Pushing it to 3,000 edge nodes spread across 60 PoPs without causing a cache stampede is another story entirely. We've learned the hard way that you never let the whole fleet converge at once. The trick is to treat Puppet like any other edge deployment: canary first, then staged ramp, with automated guardrails at every step.

Our pipeline looks like this: code gets pushed to a control repo, r10k and Code Manager sync it to Puppet Server, and then we use Puppet environments to gate progression. A change hits edge-canary first - that's 12 nodes, two per region. They run with an agent interval of 30 minutes but with a splay of up to 10 minutes so they don't thundering-herd the compiler. If those nodes stay clean for two full runs and our Datadog cache-hit ratio doesn't dip below 92%, we promote to edge-10pct (~300 nodes), then 50%, then full fleet. The whole rollout takes about 90 minutes, and that's intentional. If you're comparing orchestration models, it's worth understanding how this differs from CDN Configuration with Chef: Recipe-Based Infrastructure for Content Delivery, where converges are more often push-driven.

Don't forget to build circuit breakers into the manifests themselves. We don't just declare the desired state; we tell Puppet when not to apply it. If an edge node's load average is above 18 or its NVMe cache disk is over 85% full, we bail on the Nginx reload. A failed Puppet run is annoying, but a forced reload on an already-hot node that drops 20Gbps of traffic is a Sev1.

# modules/cdn/manifests/edge_cache.pp
class cdn::edge_cache (
  String $origin_host,
  Integer $cache_size_gb = 800,
  Array[String] $waf_rules = [],
) {
  # Circuit breaker: don't churn cache if node is hot
  if $facts['load_1m'] < 18 and $facts['cache_disk_used_pct'] < 85 {
    class { 'cdn::nginx':
      worker_processes => $facts['processors']['count'],
      cache_path       => "/data/cache levels=1:2 keys_zone=EDGE:${cache_size_gb}m inactive=7d",
      origin           => $origin_host,
      require          => Class['cdn::tuning'],
    }

    # Safely reload only when config actually changes
    File['/etc/nginx/nginx.conf'] ~> Service['nginx']
  } else {
    notify { 'puppet-circuit-breaker-tripped':
      message => "Skipping nginx converge: load=${facts['load_1m']}, disk=${facts['cache_disk_used_pct']}%",
    }
  }

  # Ensure Varnish and ATS don't fight for the same port
  ensure_resource('firewall', '110 allow cache purge', {
    dport  => 6081,
    source => $facts['cdn_trusted_purge_nets'],
    proto  => 'tcp',
    action => 'accept',
  })
}

Advanced Patterns: Drift Detection, Exported Resources, and Debugging at Scale

Once you're past the initial rollout, the real enemy is drift. Edge nodes reboot uncleanly, someone SSHes in for a "quick fix" during an incident, or a disk gets replaced and comes back unformatted. Puppet's great at converging, but you need visibility. We run puppet agent -t --noop results into our logging pipeline and alert if more than 2% of the fleet shows corrective changes for three consecutive runs. That usually means someone merged a bad Hiera value or an upstream package repo moved. Pair that with proper CDN Configuration Templating: Jinja2, HCL, and Go Templates for Edge Configs and you'll catch a surprising number of rendering errors before they ever hit the compiler.

For dynamic edge problems, exported resources are your best friend. Forget hardcoding peer IPs. When a new cache node comes online in Ashburn, it exports its @@nagios::check and @@cdn::peer resources to PuppetDB, and every other node in that PoP automatically picks up the new peer for consistent hashing. We use the same pattern for managing purge ACLs and TLS certificates - the Let's Encrypt node exports the cert, edge nodes collect it. It cuts our manual inventory tickets to zero and keeps the ring balanced without a human editing a YAML file at 2 a.m.

When things do go wrong, don't tail logs node-by-node. Use facts and PuppetDB queries to triage. puppet query 'nodes[certname] { facts.os.release.major = "8" and facts.cdn_role = "edge" and report_status = "failed" }' will tell you in five seconds if your failures are isolated to a bad OS upgrade. And if you want to prevent bad manifests from even compiling, that's where you should layer in CDN Policy as Code with OPA and Sentinel: Enforcing Rules Automatically in your CI pipeline to block manifests that, say, try to open port 22 to 0.0.0.0/0. We've saved ourselves from at least four self-inflicted outages that way.

Frequently Asked Questions