Negative Caching: Why and How to Cache Error Responses

Negative caching is the practice of storing error responses at the CDN edge so that repeated requests for missing or failed resources are served from cache rather than forwarded to the origin. While most caching discussions focus on successful responses, error responses can generate enormous origin load if left uncached. A single broken link shared on social media can produce millions of 404 requests that bypass the cache and hammer your origin, creating the same effect as a denial-of-service attack. Understanding which errors to cache, for how long, and when caching errors becomes dangerous is a critical part of any comprehensive caching strategy.

What Is Negative Caching and Why Does It Matter?

In caching terminology, a positive cache stores successful responses, while a negative cache stores error or empty responses. The term originates from DNS, where negative caching stores NXDOMAIN responses to prevent repeated lookups for nonexistent domains. In CDN contexts, negative caching applies the same principle to HTTP error responses.

Without negative caching, every request for a missing resource passes through the CDN to the origin. For high-traffic sites, this creates several problems:

• Origin overload: Thousands of 404 requests per second consume origin CPU and bandwidth for resources that do not exist.
• Error amplification: A single origin error can cascade through the CDN, with each edge PoP independently retrying the failed request.
• Wasted bandwidth: Even error responses consume bandwidth between CDN and origin, especially when custom error pages include images and styling.
• Masking real issues: When 404 traffic dominates origin logs, legitimate errors become harder to identify and diagnose.

Which Error Codes Should Be Cached?

The HTTP specification in RFC 9110 defines which response codes are cacheable by default. Not all errors should be cached, and the distinction matters for both correctness and reliability.

Cacheable Error Codes

Error Codes You Should Not Cache

Server-side errors represent transient conditions and should almost never be cached:

• 500 Internal Server Error: Indicates a bug or temporary failure. Caching this response means users see the error long after the fix is deployed.
• 502 Bad Gateway: The origin or an upstream service is temporarily unreachable. Recovery is usually measured in seconds.
• 503 Service Unavailable: Explicitly signals a temporary condition, often with a Retry-After header. The origin expects the client to retry.
• 504 Gateway Timeout: The upstream did not respond in time. Caching this punishes users for a momentary network issue.

The one exception is micro-caching server errors for one to five seconds during an active cache stampede. This prevents hundreds of edge nodes from simultaneously retrying against an already-overwhelmed origin, giving it breathing room to recover.

How Do You Configure Negative Caching TTLs?

The right TTL for a negative cache entry depends on the error type and your application's behavior. Setting it too short negates the protection benefit. Setting it too long serves stale errors after the resource becomes available.

TTL Guidelines by Error Type

404 Not Found: Use 60 to 600 seconds. For applications with frequent deployments where new URLs appear regularly, stay at the shorter end. For static content sites where URLs rarely change, the longer end is appropriate. Always set shorter negative TTLs than your positive cache TTLs to ensure that newly created resources become visible quickly.

410 Gone: Use 3,600 to 86,400 seconds. A 410 explicitly tells clients the resource has been permanently removed. A longer TTL is appropriate because the origin has deliberately signaled that this URL will not return content. This also helps search engines deindex the URL faster.

301 and 308 Permanent Redirects: Use 3,600 to 86,400 seconds. These are permanent by definition. Caching them aggressively reduces redirect-chain latency and origin load. However, leave enough room to correct a misconfigured redirect without waiting hours for the cache to expire.

Configure negative caching through Cache-Control headers on your origin error responses:

# Nginx: Cache 404 responses for 5 minutes
error_page 404 /404.html;
location = /404.html {
    internal;
    add_header Cache-Control "public, max-age=300, s-maxage=300";
}

# Nginx: Cache 410 responses for 1 hour
error_page 410 /410.html;
location = /410.html {
    internal;
    add_header Cache-Control "public, max-age=3600, s-maxage=3600";
}

How Does Negative Caching Prevent Error Amplification?

Error amplification occurs when a single origin failure produces cascading errors across the CDN. Without negative caching, the sequence is destructive:

1. Origin returns a 404 for /product/discontinued-item.
2. The CDN does not cache the error and discards the response.
3. The next request for the same URL bypasses cache and hits the origin again.
4. At 1,000 requests per second, the origin processes 1,000 identical lookups every second, each returning the same 404.
5. This load consumes database queries, application logic, and network bandwidth that should serve valid requests.

With negative caching enabled, step 2 changes: the CDN caches the 404 response. Steps 3 through 5 do not occur because subsequent requests are served from the edge cache. The origin processes one lookup instead of thousands.

This protection becomes especially valuable during link rot events. When an old blog post linking to your site spreads through social media, thousands of users may click a URL that no longer exists. Without negative caching, each click becomes an origin request. With a 5-minute negative cache TTL, only one request per 5 minutes per edge PoP reaches the origin.

How Do Custom Error Pages Interact with Negative Caching?

Many sites serve custom-branded error pages with navigation, search boxes, and suggested content. These pages can be significantly larger than a simple error text response, making negative caching even more important for bandwidth savings.

When implementing custom error pages with negative caching, consider these factors:

• Error page size: A custom 404 page with images, CSS, and navigation might be 50 KB. At 1,000 uncached requests per second, that is 50 MB/s of origin bandwidth just for errors.
• Dynamic elements: If your error page includes personalized recommendations or search suggestions, it may be marked uncacheable. Consider using static error pages at the CDN level and loading dynamic elements client-side.
• Consistency: Ensure your error pages include proper TTL headers so the CDN treats them as cacheable. Some frameworks send error responses with Cache-Control: no-store by default.

How Does Negative Caching Interact with Health Checks?

CDN health checks and negative caching operate on related but separate timelines. Health checks determine whether an origin is available for new requests. Negative caching determines how long an error response is served from edge cache. Misalignment between these systems creates confusing behavior.

Scenario 1: Origin recovers, cached errors persist. The origin was returning 500 errors. It recovers and health checks pass. But the CDN is still serving cached 500 responses until their TTL expires. Users see errors despite a healthy origin. This is why caching 5xx errors is generally discouraged.

Scenario 2: Origin fails, no cached errors. Without negative caching, each edge node independently discovers the origin failure on every request. The origin, already struggling, receives maximum traffic at its worst moment. Health checks eventually mark it down, but the damage window can be significant.

The solution is to coordinate TTLs with health check intervals. If health checks run every 30 seconds, consider negative cache TTLs that are multiples of 30 seconds. This way, the cached error expires around the same time the next health check would run, and fresh requests coincide with updated health status.

What Are Provider Default Behaviors?

CDN providers handle negative caching differently by default, and understanding these defaults prevents surprises:

CloudFront is notable for its built-in negative caching with configurable TTLs per error code. You can set different cache durations for 400, 403, 404, 405, 414, 500, 501, 502, 503, and 504 responses. Cloudflare requires explicit opt-in through Cache Rules or origin Cache-Control headers.

When Should You Avoid Negative Caching Entirely?

Negative caching is not universally beneficial. Several scenarios require careful consideration:

• Authentication endpoints: A 401 or 403 response to a specific user should never be cached and served to other users. These responses are inherently personalized. Ensure your cache key design accounts for authentication state.
• Rapid content creation: If your application creates URLs dynamically and users immediately access them, even a 60-second negative cache means users see a 404 for the first minute after creation. Use cache purging or bypass for these paths.
• API endpoints with side effects: A POST request returning a 409 Conflict should not be cached because the next request with different data might succeed. Negative caching is primarily for GET and HEAD requests.
• Debugging and development: During development, negative caching obscures real-time changes. Disable it for staging environments or use very short TTLs.

Negative caching is a simple but powerful technique that belongs in every CDN configuration. By caching error responses appropriately, you protect your origin from unnecessary load, improve response times for error pages, and prevent cascading failures during outage events. The key is choosing the right TTL for each error type and coordinating with cache hierarchy settings and health check systems to ensure cached errors expire at the right time. Combined with cache hit ratio optimization, proper negative caching rounds out a complete caching posture that handles both successful and failed responses efficiently.