BGP Routing: How the Internet Finds Its Way
BGP (Border Gateway Protocol) is the routing protocol that holds the internet together — the system that determines how data packets find their way across the global network of 75,000+ autonomous systems. Often called "the postal service of the internet," BGP enables every ISP, cloud provider, and CDN to exchange routing information and deliver traffic to its destination. For CDN providers, BGP is especially critical because it powers anycast — the technique that directs users to the nearest edge server. Understanding BGP is essential for grasping how CDNs route content globally.
What Is the History of BGP?
BGP's history reflects the internet's evolution from a research network to a global commercial infrastructure:
- Pre-BGP (1982-1989): The early internet used EGP (Exterior Gateway Protocol), which assumed a tree-shaped network topology with a single core. This model broke down as the internet grew more complex with multiple interconnected backbone providers.
- BGP-1 (1989, RFC 1105): The first version of BGP introduced the concept of path vector routing — instead of just knowing "I can reach network X," routers now knew "I can reach network X via the path A→B→C." This path information enabled smarter routing decisions and loop detection.
- BGP-2 and BGP-3 (1990-1991): Incremental improvements to the protocol, adding better error handling and support for the growing internet routing table.
- BGP-4 (1994, RFC 1771; updated in RFC 4271, 2006): The version still in use today. BGP-4 added CIDR (Classless Inter-Domain Routing) support, enabling route aggregation that prevented the routing table from growing unmanageably large. Without CIDR, the internet's routing table would have exhausted router memory by the late 1990s.
Today, the global BGP routing table contains over 1 million IPv4 prefixes and 200,000+ IPv6 prefixes, managed across 75,000+ autonomous systems.
How Does BGP Work?
BGP operates as a "path vector" routing protocol, where each network (Autonomous System, or AS) exchanges information about which IP address ranges (prefixes) it can reach and the path of ASes needed to get there:
| Metric | Edge Processing | Origin Processing | Improvement |
|---|---|---|---|
| Latency (p50) | 8–15 ms | 120–300 ms | 15–20x faster |
| Cache Hit Ratio | 85–95% | N/A | Reduces origin load |
| TLS Handshake | 1–5 ms (nearby PoP) | 50–150 ms | 10–30x faster |
| Bandwidth Cost | $0.01–0.04/GB | $0.05–0.12/GB | 3–5x cheaper |
| Availability | 99.99% (multi-PoP) | 99.9% (single DC) | 10x fewer outages |
- Peering Session Establishment: Two BGP routers (called "peers" or "neighbors") establish a TCP connection on port 179. They exchange OPEN messages containing their AS number, BGP version, and capabilities.
- Route Exchange: Peers exchange UPDATE messages containing route announcements (new or changed routes) and withdrawals (removed routes). Each announcement includes the prefix (e.g., 104.26.0.0/20), the AS-PATH (the sequence of ASes the route traverses), and various attributes (next-hop, community tags, MED).
- Route Selection: When a router receives multiple paths to the same destination, it applies a decision process (in order of priority): highest local preference → shortest AS-PATH → lowest origin type → lowest MED → prefer eBGP over iBGP → lowest IGP metric to next-hop → lowest router ID.
- Route Propagation: Selected best routes are propagated to other BGP peers, with the router's own AS prepended to the AS-PATH (preventing routing loops).
What Are the Most Notable BGP Incidents?
BGP was designed without built-in authentication, making it vulnerable to route hijacking — when an AS incorrectly announces someone else's IP prefixes:
- Pakistan YouTube Hijack (2008): Pakistan Telecom attempted to block YouTube domestically by advertising a more specific route for YouTube's IP space. Due to BGP's trust-based design, this announcement propagated globally, redirecting YouTube traffic worldwide through Pakistan's network for approximately 2 hours.
- Google Outage (2020): A misconfiguration at a Nigerian ISP caused it to advertise Google's IP prefixes to its peers. Traffic intended for Google services was routed through networks that couldn't handle it, causing outages for millions of users.
- Facebook/Meta BGP Withdrawal (October 2021): During a routine maintenance operation, Facebook accidentally withdrew all BGP routes for its IP space. Because Facebook's internal DNS servers were also unreachable (their names pointed to now-unroutable IPs), the outage lasted approximately 6 hours. Even Facebook engineers were locked out — badge access systems depended on the same network. This incident demonstrated how BGP withdrawal can create cascading failures.
- Rostelecom Hijack (2020): Russian ISP Rostelecom briefly hijacked prefixes belonging to Cloudflare, Amazon, and other major services, raising concerns about state-sponsored BGP manipulation.
How Do CDNs Use BGP for Anycast?
CDN providers use BGP in a specialized way called anycast: they announce the same IP prefix from multiple geographic locations simultaneously. When a user sends a request to an anycast IP address, BGP routing automatically directs the packet to the nearest (in BGP terms, shortest AS-PATH) location:
- Global Load Distribution: By announcing the same prefix from 200+ locations, a CDN like Cloudflare ensures that users in every region reach a nearby edge server without any application-layer routing logic. BGP handles it at the network layer.
- DDoS Absorption: Anycast inherently distributes DDoS attack traffic across all announcing locations. An attack targeting a single IP address is absorbed by the CDN's entire global network, with each PoP handling only its geographic portion of the attack traffic.
- Automatic Failover: If a CDN PoP goes down and stops advertising its BGP routes, traffic automatically shifts to the next nearest PoP within seconds — faster than DNS-based failover. This is key to CDN geo-routing strategies.
How Is BGP Being Secured?
Several technologies are being deployed to address BGP's security weaknesses:
- RPKI (Resource Public Key Infrastructure): RPKI allows IP address holders to cryptographically sign Route Origin Authorizations (ROAs) that specify which ASes are authorized to announce their prefixes. Validators can then reject unauthorized announcements. Major CDN and cloud providers (Cloudflare, AWS, Google) now sign their routes with RPKI.
- BGPsec (RFC 8205): Extends RPKI to protect not just the origin but the entire AS-PATH, preventing path manipulation. However, BGPsec requires changes to BGP processing at every AS in the path, making deployment challenging.
- Route Filtering: Network operators implement prefix filters based on Internet Routing Registry (IRR) data, rejecting announcements for prefixes they shouldn't receive. This is a practical defense against accidental hijacking but doesn't prevent determined attackers.