How to Build an Anycast Network

When you type a URL into your address bar, your browser performs a DNS lookup to resolve the human-readable domain name to an IP address necessary for networking software, for example 216.24.57.1. Armed with that IP, how do the packets comprising the request actually get from your local machine to a server somewhere in the world capable of handling that request? How do you ensure those packets route to the server that can handle the request most efficiently?

Serving traffic as efficiently as possible, regardless of where it originates, is a critical requirement at Render. One answer to this challenge is to deploy a global fleet of servers to create an anycast network. What is an anycast network? What is it good for? How can you build your own? This post will answer all of these questions, but we’ll first need some background on the building blocks of global networking.

A Brief Overview of the Internet

The Internet is the greatest decentralized network in the history of mankind. There is no central authority that controls who is allowed to connect or how you can use it. How do we manage billions of internet-connected devices without any single organization in charge of maintaining the system? This is accomplished through the coordination of many independent entities called Autonomous Systems (AS). ASs are the networking backbone of the internet. These entities, often managed by ISPs or large technology organizations, are responsible for handling traffic for the IP addresses they manage and advertising the paths to all other IP prefixes of which they are aware. Connections between ASs, also known as peers, are established through the manual configuration of connections between routers.

Once a connection has been established, ASs use the Border Gateway Protocol (BGP) to coordinate with each other. BGP carries the information necessary for ASs to determine the optimal path to a target. Many factors can be used to define "optimal", but the most common heuristic used is choosing the path with the fewest number of "hops" between ASs before reaching the destination. Each AS will store the best path for all IP prefixes in its routing table. When a route is added or removed, the AS will update its routing table, if necessary, and advertise the change to its peers so they can do the same. Each peer sends keep-alive messages to each other at regular intervals. If an AS does not send a keep-alive message before the configured timeout, the peer will stop advertising the AS as a reachable hop and packets will use a different path to reach the target if it exists.

What is Anycast?

An anycast network is two or more servers advertising the same IP address from different locations. Why would we want to do this?

Serving traffic from a single location often lacks the redundancy required for critical or latency-sensitive applications that serve users around the world. In the worst case of a global round-trip request, you can expect 300 ms of latency just in transit time. This doesn't account for the time it takes the server to process the request. Chain a few of these requests together and you're looking at a very sluggish application and a frustrating user experience. For applications that require high availability and low latency, you will want an architecture that has the following characteristics:

• Globally distributed
• Automatically routes traffic to the optimal server
• Tolerates failures of individual servers

We can leverage the advanced routing capabilities of BGP to create a fault-tolerant, low-latency system by building an anycast network. By advertising an IP from many servers around the world, we can rely on BGP to route users' requests to the nearest Point of Presence (PoP) -- or server, in other words -- that advertises the IP. When one of those PoPs fails, it will stop sending keep-alive messages to its peers, connected ASs will stop advertising the path, and traffic will reroute to a healthy PoP.

How Render Uses Anycast

When you deploy a web service or static site on Render, you always get a subdomain of onrender.com that makes your service accessible from the internet. In many cases, users want to use a custom domain that they own instead of an onrender.com subdomain. Users can create a DNS A record for their custom domain that points to an IP address provided by Render.

Render strives to create the most delightful developer experience possible. One way we do this is by making it as simple as possible to configure your DNS settings for custom domains. We provide a single IP address (our anycast IP address) to use in your DNS settings. When a request reaches one of the servers advertising that IP, we will forward it to one of possibly several origins around the world, depending on where your site is hosted and whether it's a web service or a static site.

We can't make any assumptions about where traffic will originate, and we can minimize latency by routing requests to the server closest to the end-user that's capable of responding to the request. For web services, the request always needs to go to the origin server--the server you've deployed to. But for static sites, we can rely on BGP to route requests to the server closest to the end-user; we advertise our IP from anycast servers worldwide, which inspect each request and forward it to a server capable of responding. So, the server that serves a request is one of hundreds of geographically distributed CDN edge servers containing a cached copy of your static site. If one of those servers fails, its peers will stop advertising the route and the traffic will be sent elsewhere.

Building an Anycast Network

Now that we understand the components that make up an anycast network, as well as the use cases for it, we're ready to dive into how you can build your own. At a high level, the steps for setting up anycast are:

• Obtain an Autonomous System Number
• Acquire an IP Address
• Find a Hosting Provider
• Configure BGP

Obtain an Autonomous System Number

The first step for setting up an anycast network is getting an autonomous system number (ASN). An ASN is a unique identifier that other ASs will use when announcing a path to reach you via BGP. Having an ASN is also a prerequisite for acquiring an IP address. ASNs are managed by five Regional Internet Registries. As the name suggests, each registry is responsible for managing the ASNs within specific geographic regions of the world.

Each registry has different requirements for acquiring an ASN. If you’re based in the United States, you'll work with ARIN to get yours. Their process is straightforward. You just need to submit a request with some supporting documentation along with an explanation of why you need the ASN. ARIN grants ASNs to organizations for two purposes:

1. Unique Routing Policy: Organizations who want to define routing policies that differ from their connected peers.
2. Multi-homed: Organizations that will be peering with multiple providers. Multi-homing is usually done to improve coverage and resiliency to failures of individual providers.

When Render applied for an ASN, we wanted the ability to advertise from multiple providers, so we used multi-homing as a justification for receiving an ASN. After submitting our application, ARIN responded with our ASN within a week.

Acquire an IP Address

Now that you have an ASN, you can acquire an IP block to advertise. The minimum IP prefix that can be advertised is a /24 for IPv4 and a /48 for IPv6. This limit exists to prevent the routing tables that ASs need to manage from growing too large.

You are going to need to decide whether you want to buy or lease your IP block. The dynamics at play here are similar to buying and leasing most things. Buying an IP block comes with a large upfront cost, while leasing requires less capital at the start but you will be paying a fee for as long as you use the IP block. Leasing also comes with additional terms and conditions on how you can use the IP block. If you have the capital and intend to use the IP block for an extended period of time, it likely makes sense to buy. Almost all available IPv4 addresses have been allocated and demand for them continues to rise. You can expect the cost to obtain them to continue to grow until IPv6 adoption is widespread enough to justify dropping support for IPv4.

Since we knew that we would need the IP block for the foreseeable future, we decided to buy one. Asking all of our users to update their DNS records would be extremely disruptive, so we wanted to ensure that our IP block would remain stable for as long as it was in use. We had a good experience using IPv4 Global to purchase our IP block. The remainder of this post will assume that you have chosen to buy instead of lease.

There are a couple of things you should verify before buying an IP block. The block needs to be registered with the Regional Internet Registry that you acquired your ASN from, or it must be transferable to them. You will also want to verify that the IP addresses in the block are in good standing. Some IP addresses have been used for nefarious purposes and end up on widely-used blocklists. There are many free services available that will check an IP address against the most popular databases of bad IPs.

Once you purchase your IP block, you will need to go through the process of transferring ownership from the previous holder of the IP block. The service you use to purchase the block will likely guide you through that process, which will involve working with your Regional Internet Registry to transfer ownership. The exact steps will depend on the registry and whether the block needs to be transferred between registries. You can look at ARIN's documentation to get an idea of what's involved in the process.

Find a Hosting Provider

Now that we have an IP address to advertise, it's time to determine what providers we will use to advertise it. Not all providers support BGP, so you're going to want to find ones that have the capability. In our anycast network setup process, we decided to go with Vultr and Equinix Metal.¹ The latter is a bare metal provider rather than a VPS provider, so you should verify whether that makes sense for your use case before deciding to go with them. We chose Vultr because they had global coverage, low cost, and thorough documentation on how to set up BGP. You can find discussions comparing VPS providers on forums like LowEndTalk to determine what provider makes sense for you.

Configure BGP

The next step is to set up your server and start accepting traffic for your anycast IP address. After getting your application running on the server, you can get started on configuring BGP.

When we configured BGP, we first needed to submit an application to Vultr requesting that they enable BGP for our account. They had us send a Letter of Authorization that allowed them to advertise our IP address from their platform. As a part of this process, Vultr created a record in the Internet Routing Registry on our behalf that signals to the rest of the internet how to handle traffic for our IP. It's a great UX to have Vultr take care of this for you. However, it’s important to note that Vultr will delete this record if you end up migrating away from them. You must create a new record before they delete their copy to avoid any service disruptions.

The next step is configuring BGP on your server. You're going to need to choose a BGP daemon to handle the advertisement. The two most common are Quagga and BIRD. We decided to use BIRD since Vultr recommended it and they have excellent documentation for getting it running.

After installing BIRD, create a file called /etc/bird.conf with the following contents:

router id 203.0.113.123 # IP of the Vultr server running BIRD

protocol bgp vultr
{
    local as 123456; # your ASN
    source address 203.0.113.123;
    import none;
    export all;
    multihop 2;
    neighbor 169.254.169.254 as 64515; # Vultr’s peering information
    password "my-secret-password"; # BGP password provided by Vultr
}

After restarting BIRD, your server will establish a peering session with Vultr’s AS. At this point, your server is capable of transmitting packets with the rest of the internet. Now you just need to tell the rest of the internet that you’re here! The final step is to announce your route. Append the following to the same bird.conf file:

protocol static
{
        route 123.4.567.890/24 via 203.0.113.123; # your IP via Vultr’s IP
}

protocol device
{
        scan time 5;
}

Restart BIRD again to load the new configuration. At this point, your BGP server is announcing the route to Vultr, who is in turn announcing the route to their peers (who then announce it to their peers). The process continues until your new route propagates across the internet. Your server can now serve traffic for your IP! Repeat these same steps with a second server in another region and you have your very own anycast network.

Conclusion

While this post provides the information needed to get your anycast network running, you may still need to adjust your settings to fine-tune the traffic patterns for your network. As mentioned previously, ASs usually choose the path with the fewest number of hops to a destination. Fewer hops does not necessarily mean it will be faster if, for example, one of those hops crosses the Atlantic Ocean. If you notice suboptimal performance, you may want to look into changing the traffic patterns with tools like communities and prepending.

Anycast networks are a powerful tool that leverages the networking infrastructure of the internet to create resilient and performant systems. I hope this post was insightful enough to add another tool to your toolbelt when building reliable systems.

1. We don't use Vultr or Equinix Metal anymore because of how our Anycast needs have evolved; we'll discuss our new setup in another blog post! ↩