Origin Request Prioritization: Queuing and Scheduling for Critical Content

Back in 2019, we pushed a routine purge for a major retail customer right before their flash sale, and the origin collapsed in under 90 seconds. It wasn't traffic volume — it was ordering. Ten thousand image revalidation requests choked the origin queue ahead of the checkout API and HTML shell, so users saw 5s TTFB on critical pages while thumbnails loaded just fine. That incident forced us to stop treating the origin queue as a dumb FIFO pipe and start treating it like a scheduler with opinions about what's critical.

Why FIFO Origin Queues Fail When Traffic Isn't Uniform

Most CDN origin shields start with the simplest possible queue: first in, first out. It’s easy to reason about, fair on average, and it works until you’ve got mixed content types fighting for the same constrained origin connections. In practice, you never have uniform traffic. You’ve got HTML documents and API responses that block rendering, alongside images, fonts, beacons, and background prefetch that don’t. When you push all of them through one FIFO, you’re guaranteeing head-of-line blocking for the stuff users actually wait on. I’ve seen a single bot scraping 40k legacy image URLs starve checkout for three minutes because the queue had no concept of critical content.

Request prioritization isn’t just about slapping a priority label on a request. It’s about understanding what happens at the origin when concurrency is capped, which it always is. Your Origin Health Monitoring Patterns: Active Probes, Passive Checks, and Hybrid Approaches setup might show the origin as healthy — 200ms average response, low error rate — but that average hides what's really happening. Your P50 might be fine while your P95 for critical content spikes to 2 seconds because non-critical requests are occupying pooled connections. Per the HTTP Working Group Specifications, HTTP/2 and HTTP/3 give you stream prioritization on the client-to-edge hop, but that signal rarely survives to the edge-to-origin hop unless you explicitly propagate and act on it.

We learned to classify origin fetches into at least three tiers before they ever hit the queue, and we enforce that classification at the edge, not at the origin. That way we don’t rely on the origin to tell us what’s important after it’s already overloaded.

// Origin request classifier - runs at edge before enqueue
// Priority is assigned based on content criticality, not arrival time

function classifyOriginRequest(req) {
  // Tier 0: Blocks rendering or transaction - never queue behind Tier 1/2
  if (req.routeType === 'document' || req.routeType === 'api') {
    return { tier: 0, weight: 100, timeoutMs: 800, retryBudget: 2 };
  }
  // Tier 1: Needed for above-the-fold but can be stale-while-revalidate
  if (req.cacheStatus === 'stale' && req.contentType.match(/^image|font|css/)) {
    return { tier: 1, weight: 10, timeoutMs: 1500, retryBudget: 1 };
  }
  // Tier 2: Prefetch, beacons, background sync - sheddable under pressure
  if (req.headers['purpose'] === 'prefetch' || req.isBackground) {
    return { tier: 2, weight: 1, timeoutMs: 3000, retryBudget: 0, sheddable: true };
  }
  return { tier: 1, weight: 10, timeoutMs: 1500, retryBudget: 1 };
}

// Enqueue with priority - higher weight dequeues first
originQueue.enqueue(request, classifyOriginRequest(request));

Scheduling for Critical Content Without Starving Everything Else

Once you’ve classified, you need a scheduler that actually respects those tiers under contention. A strict priority queue is tempting — always drain Tier 0 before Tier 1 — but it’ll starve your Tier 2 indefinitely during a sustained incident, and that creates its own problems. We’ve had cases where starved image revalidations built up so much stale content that we triggered a thundering herd later. What you want is weighted fair queuing or a deficit round-robin approach where critical content gets, say, 80% of origin concurrency but non-critical still makes progress.

Our current implementation uses a two-level scheduler: an admission gate and a weighted dequeuer. The admission gate sheds load *before* queuing when origin concurrency is saturated, and it sheds from the bottom tier first. If origin connections are at 95% and a Tier 2 prefetch arrives, we don’t queue it at all — we serve stale or return 202 and let the client retry. That’s where Origin Connection Pooling Optimization: Reducing TCP Overhead at CDN Scale matters, because your pool size dictates your scheduling quantum. With 100 origin connections and a 10ms dequeue interval, you’re making 100 scheduling decisions per second; get that wrong and your queue scheduling logic doesn’t matter.

The other detail folks miss is priority inheritance for collapsed requests. If 500 users request the same critical HTML shell while it’s already being fetched from origin, you’ve collapsed them into one origin request. That one request now represents 500 users’ latency, so its priority should reflect that fan-out. We bump its weight proportionally to coalesced count, otherwise a single background image fetch with weight 10 could jump ahead of a collapsed document that’s blocking half your traffic. Guidance from web.dev Articles on prioritizing Largest Contentful Paint and interaction-critical resources maps directly here — if it blocks LCP, it shouldn’t wait in the origin queue.

# tiered origin scheduler config (pseudo-envoy / nginx-style)

origin_scheduler:
  max_origin_concurrency: 120
  dequeue_interval: 10ms
  tiers:
    - name: critical
      tier: 0
      weight: 80          # 80% of dequeues under contention
      max_queue_depth: 2000
      queue_timeout: 800ms
      coalesce_weight_boost: true  # +1 weight per 10 coalesced clients
    - name: normal
      tier: 1
      weight: 15
      max_queue_depth: 8000
      queue_timeout: 1500ms
      serve_stale_on_shed: true
    - name: background
      tier: 2
      weight: 5
      max_queue_depth: 2000
      queue_timeout: 3000ms
      shed_threshold: 0.85  # start shedding Tier 2 at 85% concurrency
      shed_action: serve_stale_or_202

  # Prevent strict starvation - guarantee at least 1 dequeue per second per tier
  starvation_prevention:
    enabled: true
    min_dequeues_per_sec_per_tier: 1

What We Actually Measured After Switching From FIFO

We rolled this out on a mid-tier PoP that handles about 45k RPS to origin at peak, with a cache hit ratio around 92%. Under normal conditions you won’t see much difference — FIFO looks fine when you’re at 30% origin concurrency. The win shows up when origin slows down or when you’re absorbing a partial outage. During our last origin latency regression (origin P50 jumped from 110ms to 480ms for 8 minutes), the prioritized scheduler kept checkout API P95 under 900ms while FIFO in our canary region spiked to 3.1s. That’s the difference between users completing purchases and abandoning.

You’ve also got to pair queue scheduling with a sensible retry budget, otherwise retries become high-priority noise that amplifies the problem. We cap retries per tier and apply jittered backoff in the queue itself, not just at the client. If you haven’t tuned that, read Origin Retry Strategies for CDN Resilience: Backoff, Jitter, and Budget Limits before you touch priority weights — we wasted two weeks tweaking weights when our real issue was Tier 1 retries hammering the origin 3x per failure.

Here’s what the before/after looked like on that same PoP during the 8-minute degradation window. Both groups saw identical origin latency at the source; only the queue scheduling differed.

Implementing Priority Queuing at the Edge: What Actually Works in Production

Moving from theory to production is where most priority queuing projects stall. You don't need a complete rewrite of your origin shield — you need a deterministic classifier and a bounded queue. We've had the best luck running classification at the edge POP before the request ever hits the origin queue. That keeps origin logic simple and avoids burning origin CPU on triage. Our current setup uses three tiers: P0 for critical HTML and auth/API calls, P1 for stale-while-revalidate and prefetch, and P2 for bulk image optimization and bot traffic. It's not fancy, but it maps 1:1 to business impact.

The trick is keeping the queues bounded and independently throttled. We cap P0 concurrency at 70% of total origin connections, P1 at 25%, and P2 at 5%, with weighted fair dequeuing every 10ms. If P0 is empty, its capacity spills to P1 — you never want idle origin connections while low-priority work waits. We also set hard queue timeouts: 50ms for P0 before we attempt Origin Request Hedging: Sending Parallel Requests to Cut Tail Latency to a secondary origin, and 800ms for P2 before we just serve stale or a 202. Without those caps, a slow origin will let your P2 queue grow to 40k+ deep and OOM your shield node. We've seen it happen in under 90 seconds during a cache purge storm.

Here's the stripped-down classifier and queuing logic we run in our OpenResty/Lua shield layer. It’s not copy-paste production code, but it's close to what we shipped:

-- priority classifier + bounded WFQ enqueue (Lua / OpenResty)
local PRI_MAP = { P0 = 1, P1 = 2, P2 = 3 }
local QUEUE_LIMITS = { [1]=5000, [2]=10000, [3]=20000 }
local CONCURRENCY_WEIGHTS = { [1]=70, [2]=25, [3]=5 }

function classify_request(r)
  if r.uri:match("^/api/checkout") or r.uri:match("^/auth") then
    return PRI_MAP.P0
  end
  if r.headers["x-purpose"] == "prefetch" or r.is_stale_revalidate then
    return PRI_MAP.P1
  end
  if r.headers["user-agent"]:match("bot") or r.uri:match("/_next/image") then
    return PRI_MAP.P2
  end
  return PRI_MAP.P0 -- default to critical, fail safe
end

function enqueue_origin(req)
  local pri = classify_request(req)
  local q = queues[pri]

  if q:size() >= QUEUE_LIMITS[pri] then
    if pri == PRI_MAP.P2 then return serve_stale_or_drop(req) end
    -- P0/P1 shed: fail fast, let edge retry or hedged request win
    return ngx.exit(503)
  end

  -- WFQ dequeue tick runs every 10ms, weighted by CONCURRENCY_WEIGHTS
  q:push(req, { priority = pri, enqueued_at = ngx.now() })
  metrics:incr("origin.queue.enqueued", { pri = pri })
end

Advanced Patterns and Troubleshooting When Priorities Go Wrong

Once you've got basic tiers running, the next failure mode is priority inversion. That’s when a P0 request depends on a P2 dependency behind the scenes — like an HTML document that blocks on a slow, low-priority A/B config fetch at origin. Your queue says P0 is critical, but origin threads are still stuck waiting on P2. We fixed this by tagging downstream sub-requests with the parent's priority and by never letting P2 share the same origin connection pool as P0. Separate pools with separate Origin Timeout Configuration Guide: Connect, Read, and Total Timeout Tuning are non-negotiable. Our P0 pool uses a 300ms connect timeout and 2s read timeout, while P2 can tolerate 1s connect and 8s read because it’s not user-blocking.

Another pattern worth stealing is adaptive re-prioritization. Static rules don't survive a flash sale. We export queue depth, wait time, and origin p99 per priority to our control plane every 5 seconds. If P0 wait time exceeds 35ms for more than 30 seconds, we automatically demote 50% of P1 to P2 and shed P2 entirely. That buys back about 120-180ms of p95 for P0 during origin stress without manual intervention. And please, don't terminate TLS at your shield and then use plain HTTP to origin to "save latency" — we've debugged too many incidents where that introduced silent content injection. Keep Origin SSL Verification at CDN Edges: Preventing Man-in-the-Middle Attacks enabled even for internal origins, and pin your origin certs if you're running your own shield tier.

If you're troubleshooting, start with three metrics: queue wait time histogram per priority, origin concurrency utilization per pool, and shed/drop rate. If P0 wait time looks great but P0 origin latency is climbing from 80ms to 450ms, you've likely got origin contention, not a queuing problem — your origin's DB or upstream is saturated. If P2 shed rate is 0% but P2 queue depth is constantly pegged at its limit, your limit is too low and you're churning. The most common misconfiguration we see is setting queue limits in counts instead of bytes or time. 5,000 queued API POSTs is a very different beast than 5,000 queued 4K image fetches.

Frequently Asked Questions