CDN Vendor Evaluation Matrix: Scoring Providers on What Actually Matters

Two years ago I sat in a room while a retailer picked a CDN because their deck showed 310 Tbps of edge capacity and 165 PoPs. Six weeks after signing, their team was waiting 45 seconds for cache purges to propagate and couldn't get raw logs for over four hours. The problem wasn't the vendor's network — it was the evaluation sheet they'd used. It gave 25% weight to PoP count and 5% to purge speed and observability. If you've ever watched a scorecard declare a winner that your engineers didn't trust, you know exactly how that story ends.

Why Feature Checklists Give You False Confidence

Most teams start their CDN comparison with a spreadsheet that looks responsible: 60 rows of features, a simple Yes/No or 1-5 rating, and an average at the bottom. I've reviewed dozens of these sheets over the last decade, and they almost always declare a tie or favor the cheapest bidder. The issue isn't effort — it's that a binary checklist treats "supports stale-while-revalidate" as equal to "has Anycast with automated failover in your primary markets." They're not equal, and your users will feel the difference on the first traffic spike.

Back in 2019, we ran an RFP for a global video platform and made this exact mistake. Every vendor checked the boxes for HTTP/2, HTTP/3, Brotli, image optimization, and WAF. On paper, they were identical. But when we dug into how they handled cache key customization and origin shielding in Asia-Pacific, the gaps were massive. One provider only allowed cache keys based on URL, while another let us include headers, cookies, and query strings with precise control — critical for our personalized manifests. We'd have missed that if we hadn't moved beyond Yes/No scoring. If you're starting your process, your CDN RFP Template: Writing Requirements That Get Meaningful Vendor Responses needs to force vendors to explain how they do something, not just that they do it.

That's where a weighted evaluation matrix fixes things. Instead of treating every requirement as equal, you assign weights that reflect business impact and test them with evidence — not slideware. You score what you can actually measure during a trial, not what the sales engineer promises. The structure below is what I use now for every mid-market and enterprise evaluation. It's simple but it prevents popularity contests.

// Example: Weighted scoring model - weights must total 1.0
{
  "evaluation_matrix": {
    "performance": {
      "weight": 0.30,
      "criteria": {
        "p95_ttfb_global": 0.40,
        "cache_hit_ratio": 0.35,
        "origin_offload_percent": 0.25
      }
    },
    "reliability_ops": {
      "weight": 0.25,
      "criteria": {
        "purge_propagation_p95": 0.30,
        "config_deployment_time": 0.25,
        "log_delivery_delay": 0.25,
        "api_uptime_sla": 0.20
      }
    },
    "security": { "weight": 0.20 },
    "developer_experience": { "weight": 0.15 },
    "commercial": { "weight": 0.10 }
  },
  "scoring": "score = Σ (criterion_score_0_to_5 * criterion_weight) * category_weight"
}

Building Weights That Match Your Traffic, Not Their Marketing

You can't set weights in a vacuum. A media site serving 80% cacheable video has completely different priorities than a SaaS app with 70% uncacheable API calls. I always start by pulling the last 30 days of origin logs and classifying traffic: cacheable vs. dynamic, geographic distribution, and request mix. One e-commerce client I worked with discovered 62% of their origin load was just three API endpoints for cart and inventory — suddenly edge compute and origin shield architecture mattered more than raw throughput. Without that data, they'd have over-weighted PoP count and under-weighted shielding.

Get your stakeholders to argue about weights before you see any vendor demos. I run a workshop where product, engineering, SRE, and security each get 100 points to allocate across categories. Engineering might push for fast config deploys and Terraform provider maturity, while security wants WAF rule propagation under 30 seconds and detailed audit logs. When we did this for a fintech migration, the SRE team's 35-point allocation to observability completely changed our final ranking. Vendors who looked great on performance alone dropped when we scored log delay and alerting flexibility. Your CDN Proof of Concept Planning: Designing a PoC That Answers Real Questions should then test those weighted priorities — don't run a generic PoC that measures everything equally.

Keep your scale consistent and evidence-based. I use a 0-5 scale where 0 is "doesn't exist or failed test," 3 is "meets requirement with manual workarounds," and 5 is "exceeds requirement and we verified it in PoC." Every score above a 3 needs a link to a test result, HAR file, or API response. That discipline stops vendor scoring from becoming a beauty contest. The small script I use to calculate final scores also flags where vendor claims don't match PoC data, so you can address gaps before contract stage.

# weighted_score.py - Calculate provider score from PoC results
def score_provider(results):
    # results = {category: {criterion: {score, weight}}}
    total = 0.0
    for cat, cat_data in results.items():
        cat_weight = cat_data["weight"]
        cat_score = sum(
            c["score"] * c["weight"] for c in cat_data["criteria"].values()
        )
        total += cat_score * cat_weight
        print(f"{cat}: {cat_score:.2f} * {cat_weight} = {cat_score * cat_weight:.2f}")
    return round(total, 2)

# Example: Vendor A scored 4.2 on performance but 2.8 on ops
# weighted total = 3.61 vs Vendor B at 3.88 (more balanced)
# The math makes tradeoffs visible - you can't hide weak ops with strong marketing

The Five Measurement Areas Vendors Can't Spin

Marketing sheets love to highlight aggregate network capacity and number of PoPs — numbers that rarely correlate with your users' experience. What matters is how a provider performs for your specific workload, under your configuration, in your key geographies. I focus on five areas and demand real tests for each. For performance, we don't just measure average TTFB; we look at p95 and p99 TTFB from real user locations using RUM or synthetic tests from Catchpoint or ThousandEyes. The guidance in the HTTP Working Group Specifications around caching semantics and the cache-control directives is what determines whether that performance is repeatable, not just fast once. And articles from web.dev Articles on LCP and cache efficiency helped us tie CDN behavior directly to Core Web Vitals for that retailer.

Operational behavior is where you'll live day-to-day, and it's what most teams forget to score heavily. How long does a purge take to reach 99% of edge nodes? Can you deploy a config change via API in under 60 seconds and roll it back instantly? How quickly do logs arrive in your S3 bucket or Splunk — is it 5 minutes or 90 minutes? These aren't nice-to-haves; they determine your incident response time. I've seen vendors claim "instant purges" that actually took 12-18 seconds at p95 during our PoC, which broke our content update workflow. That's why we test this under load, with our actual cache keys and surrogate keys. Your final cutover plan in the Comprehensive CDN Migration Checklist: Every Step from Planning to Cutover depends on these numbers being honest, so score them before you sign.

// Example weighted scoring model - keep it in code, not just Excel
const evaluationMatrix = [
  { criterion: "p95 TTFB (global)", weight: 0.25, vendorA: 4, vendorB: 3 },
  { criterion: "Cache Hit Ratio", weight: 0.20, vendorA: 5, vendorB: 4 },
  { criterion: "Origin Shield Cost & Effectiveness", weight: 0.15, vendorA: 3, vendorB: 5 },
  { criterion: "WAF & DDoS Mitigation (p95 mitigation time)", weight: 0.20, vendorA: 4, vendorB: 4 },
  { criterion: "Real Support Response (sev-1)", weight: 0.10, vendorA: 2, vendorB: 5 },
  { criterion: "Contract & Egress Pricing ($/TB)", weight: 0.10, vendorA: 5, vendorB: 3 },
];

function weightedScore(vendorKey) {
  return evaluationMatrix.reduce((total, row) => {
    return total + (row[ vendorKey ] * row.weight);
  }, 0).toFixed(2);
}

// weightedScore('vendorA') => 3.95
// weightedScore('vendorB') => 3.85  // Cheaper isn't always better when support is a 2