CDN Configuration with Chef: Recipe-Based Infrastructure for Content Delivery

At 2:47 AM last winter our on-call channel lit up because a single PoP in Frankfurt was serving three-week-old JavaScript while the rest of the fleet was perfectly fine. Turns out someone had SSH'd in months earlier, hand-edited a Varnish VCL file to hot-fix a regex, and never committed it. A routine kernel patch rebooted the box and wiped it. That's the moment my team finally stopped treating CDN edge config as a craft project and started treating it as code with Chef.

Why Chef's Convergence Model Fits CDN Edge Fleets

I've managed CDN infrastructure long enough to see every trend cycle through — from bash scripts copied via rsync to shiny new declarative tools. What keeps me coming back to Chef for CDN work isn't nostalgia, it's the convergence model. You don't tell Chef "run this command," you tell it "this box should look like this," and it figures out how to get there or does nothing if it's already there. When you've got 80 or 300 edge nodes, that idempotency isn't a nice-to-have, it's how you sleep at night.

Back in 2018 we tried to manage our ATS and Nginx edge layer with a series of shell scripts triggered by Jenkins. It worked until it didn't. One script would set cache-control headers but wouldn't clean up old ones, so we ended up with duplicate directives and weird behavior that only showed up under high load. Chef's resources fixed that because they are declarative. If I say a file should contain a specific header template and a service should be running, Chef will enforce that state every 30 minutes, not just once. That periodic enforcement catches drift, and drift is the silent killer for content delivery.

If you're evaluating other tooling, it's helpful to compare approaches. Many teams I've talked to are weighing CDN Automation with Ansible Playbooks: Configuration Management at Edge Scale for push-based simplicity, or looking at CDN Infrastructure with Pulumi: Type-Safe CDN Configuration in Real Programming Languages when they want real language features. Chef sits in a different spot — it's agent-based and excels when you need continuous enforcement on long-lived edge servers where you can't afford configuration rot.

# recipes/edge_varnish.rb - Enforce Varnish + VCL state on all CDN edges
node.default['cdn']['edge']['ttl'] = 86400
node.default['cdn']['edge']['stale_while_revalidate'] = 60

package 'varnish' do
  version node['cdn']['varnish']['version']
  action :install
end

template '/etc/varnish/default.vcl' do
  source 'default.vcl.erb'
  owner 'root'
  group 'root'
  mode '0644'
  variables(
    origin_host: node['cdn']['origin']['host'],
    ttl: node['cdn']['edge']['ttl'],
    cache_exceptions: node['cdn']['edge']['bypass_patterns']
  )
  notifies :restart, 'service[varnish]', :delayed
  # Validates syntax before Chef converges - prevents bad VCL from taking down the edge
  verify 'varnishd -C -f %{path}'
end

service 'varnish' do
  action [:enable, :start]
  supports restart: true, reload: true
end

Writing Recipes That Don't Break Your Cache Logic

The hardest part of writing Chef for CDNs isn't installing packages, it's modeling cache behavior correctly. I learned this the hard way when a junior engineer set a default TTL of 7 days in a cookbook attribute and it propagated to our API edge that was supposed to be 30 seconds. We didn't catch it because the recipe was "working" — it was just working with the wrong intent. Now we isolate cache policies by role and environment, and we test them like application code.

We structure our cookbooks so that origins, shielding, and edge caching each have their own attribute files and recipes. That way a change to image optimization headers doesn't accidentally touch the logic for HTTP Working Group Specifications compliance on surrogate keys or stale-while-revalidate. We use Chef environments for prod vs. staging and roles for `cdn-edge-us`, `cdn-edge-eu`, `cdn-origin-shield`. Foodcritic and Cookstyle run in CI, but more importantly we run Test Kitchen with InSpec to spin up a container, converge the recipes, and then curl the local edge to assert the right `Cache-Control`, `CDN-Cache-Control`, and `Vary` headers come back. If you aren't testing headers, you aren't testing your CDN.

Another pattern that's saved us is using data bags and vaults for secrets instead of hardcoding certificates or origin tokens in recipes. And for anything that touches header logic we reference web.dev Articles to double-check how browsers actually interpret those headers before we codify them. Chef makes it easy to roll that out incrementally with versioned cookbooks — you can pin 90% of the fleet to the last known good version and canary the new recipe to one PoP first.

# recipes/edge_headers.rb - Manage cache headers consistently
# Attributes are separated per content type to avoid global TTL mistakes
template '/etc/nginx/conf.d/cache_headers.conf' do
  source 'cache_headers.conf.erb'
  variables(
    static_assets_ttl: node['cdn']['cache']['static_ttl'], # e.g., 1y for immutable
    api_ttl: node['cdn']['cache']['api_ttl'],               # e.g., 30s
    html_ttl: node['cdn']['cache']['html_ttl']              # e.g., 300s + SWR
  )
  notifies :reload, 'service[nginx]', :delayed
end

# Ensure origin pull certificate is present without exposing it in git
chef_vault = chef_vault_item('cdn', 'origin_certs')

file '/etc/nginx/certs/origin.pem' do
  content chef_vault['certificate']
  owner 'root'
  mode '0600'
  sensitive true
  notifies :reload, 'service[nginx]', :delayed
end

service 'nginx' do
  action [:enable, :start]
end

# InSpec test snippet (test/integration/default/headers_spec.rb)
# describe http('http://localhost/static/app.123.css') do
#   its('headers.Cache-Control') { should match /max-age=31536000, immutable/ }
# end

How We Measure Recipe Convergence Across a Global Fleet

You can't improve what you don't measure, and Chef gives you surprisingly good telemetry if you hook it up. We ship every chef-client run to Datadog and ELK — run duration, resource updates, failures — and we alert on more than just failed runs. A run that suddenly takes 4x longer often means a template is thrashing or a service is failing to reload cleanly, which is an early sign your edge is about to flap.

When we first rolled Chef out to 120 nodes, our mean convergence time was ugly and our drift was constant. After six months of refining recipes, cleaning up community cookbooks we didn't need, and moving long-running execs to proper resources, the numbers stabilized. We also started comparing our model to what teams do with CDN Resources in CloudFormation: AWS-Native Infrastructure as Code for Content Delivery when they are all-in on AWS — Chef still wins for us on bare metal and hybrid PoPs where we need that persistent agent.

The table below is from our actual rollout last year across 214 edge nodes in 14 PoPs. It's not a lab benchmark, it's production averages over 30 days before and after we standardized everything through Chef and Test Kitchen. Your numbers will vary, but this is the kind of delta that convinces finance and operations at the same time.

Putting It Together: Deploying Chef Recipes to Live Edge Clusters

Once your cookbooks pass Test Kitchen and Cookstyle, the real challenge is getting them onto 800+ edge nodes without tanking your hit ratio. We don't push to the whole fleet at once — that's how you turn a typo into a global 502. Instead, we use Policyfiles pinned to environment-specific version constraints and roll out in waves. Our standard promotion path is staging (12 nodes in Frankfurt) → canary (5% of prod, roughly 40 nodes spread across 3 POPs) → 50% → 100%. Each wave has a mandatory 20-minute bake time where we watch cache hit ratio, origin fetch rate, and 5xx errors before promoting.

Attribute precedence is where most teams shoot themselves in the foot. You'll want to keep defaults in your wrapper cookbook, override only at the Policyfile or environment level, and never rely on role attributes for anything cache-critical. For secrets and TLS materials, we use chef-vault with short-lived certs rotated every 24 hours — no hardcoded certs in templates, ever. And if you're generating VCL or Nginx configs from attributes, don't reinvent templating logic from scratch. We've standardized on ERB for Chef but lean heavily on patterns from CDN Configuration Templating: Jinja2, HCL, and Go Templates for Edge Configs to keep our templates idempotent and reviewable.

# cookbooks/cdn_edge/recipes/varnish_cache.rb
template '/etc/varnish/default.vcl' do
  source 'default.vcl.erb'
  owner 'root'
  group 'root'
  mode '0644'
  variables(
    ttl: node['cdn']['cache']['ttl'],
    grace: node['cdn']['cache']['grace_period'],
    backend_hosts: node['cdn']['origin']['backends'],
    shard_key: node['cdn']['edge']['shard_key']
  )
  # Only reload if syntax is valid - prevents fleet-wide Varnish crash
  verify 'varnishd -C -f %{path}'
  notifies :reload, 'service[varnish]', :delayed
end

service 'varnish' do
  action [:enable, :start]
  supports reload: true, restart: true
end

# Guard against thundering herd on cold restart
execute 'warm-cache-after-converge' do
  command '/usr/local/bin/warm_cache.sh --limit 500rps'
  only_if { ::File.exist?('/var/run/varnish_restarted') }
  action :run
end

Advanced Patterns and When Convergence Fails at 3 AM

Chef's idempotency is great until it isn't. The classic failure we see is a recipe that always notifies a service restart because a timestamp or a random ID gets rendered into the config file every run. That means every 30-minute chef-client interval, Varnish does a full restart, drops its entire in-memory cache, and hammers origin. We fix this by using not_if and only_if guards aggressively and by ensuring templates only change when meaningful variables change. Use :delayed notifications instead of :immediately and batch your reloads. On a busy POP serving 180k req/s, the difference between a 200ms reload and a 4-second restart is millions of cache misses.

When convergence does go sideways, don't just SSH into one node and tail the log. Our chef-client runs are instrumented with a custom handler that pushes convergence duration, resource update count, and failure reason to Datadog. If convergence time spikes from our baseline of 8-12 seconds to over 45 seconds, or if more than 2% of nodes fail to converge in a 15-minute window, PagerDuty fires. Common culprits? Splay time misconfiguration causing all nodes to converge simultaneously and overload the Chef Infra Server, or a stuck http_request resource waiting on an unreachable origin health check. We set splay to 180 seconds and enforce a 300-second timeout on all remote resources.

For truly critical edge logic — like header manipulation or WAF rules — we don't trust convergence alone. We've implemented a post-convergence validation handler that curls localhost with a synthetic request and checks for expected headers (e.g., X-Cache-Status: HIT). If the canary nodes fail validation or if the hit ratio drops below 92% for 5 minutes post-deploy, the pipeline auto-rolls back to the previous Policyfile revision. For teams looking to harden this further, integrating policy checks like those in CDN Policy as Code with OPA and Sentinel: Enforcing Rules Automatically as a pre-convergence gate has saved us from pushing overly permissive ACLs more than once.

Frequently Asked Questions