How we built a DOH CDN with 20+ global edge servers in 10 days.

Just months ago the Commons Host static hosting CDN launched with a single edge server. Today there are over 20 edge servers around the world. The majority are inexpensive ARM-based micro servers hosted by volunteer contributors on commodity Internet connections, often Gigabit fibre. Others are virtual machines in cloud data centres which offers similar performance.

Illustration: Map of Commons Host CDN edge servers (live & WIP)

Because We Can

Kenny and I worked diligently on deployment automation. This allowed scaling the edge server count from single to double digits.

With these tools in place, we decided to build and deploy a completely new service in parallel on the same edge server network.

We chose to implement DNS over HTTPS, or DOH for short. DNS resolving is perfectly suited to the advantages and constraints of the Commons Host server network. Low latency due to global coverage, and minimal hardware requirements.

DNS, meet HTTP. HTTP, meet DNS.

Building an HTTP CDN requires learning about the Domain Name System (DNS). DNS is ancient by Internet norms; many years older than the World Wide Web or HTTP.

Standards like HTTP or DNS are the work of the Internet Engineering Task Force. This organisation provides an open, vendor-neutral discussion platform through public mailing lists. IETF also runs conferences, held 3 times yearly, rotating through the Americas, Europe, and Asia.

Photo: A session at the IETF 100 conference

At the 100th IETF conference, in Singapore where we live, a draft called DNS-over-HTTPS was presented and intensely debated. Attendees packed the conference hall. This was a meeting of worlds between DNS and HTTP experts. Even DOH's authors are respected leaders from both DNS (Paul Hoffman, ICANN) and HTTP (Patrick McManus, then Mozilla now Fastly).

I was fortunate to attend IETF 100 last year. The humbling experience left a deep impression. Implementing DOH would also be a personal tribute to this community.

How Hard Can DOH Be?

HTTP servers exist. DNS servers exist. So we just duct tape the two together? Well, basically, yes.

Driven by curiosity, Kenny wrote the first DOH implementation while brushing up on Node.js and reading the DOH draft specification, tabula rasa.

Over the next few days we rewrote and refactored the code. In the end we built a middleware called for Node.js web servers called Playdoh.

Screenshot: Playdoh GitHub repository

Playdoh relays raw UDP messages between a DOH client like Firefox and a traditional DNS server. Playdoh is 150 lines of DOH duct tape, with 300 lines of tests to make sure it sticks.

Deploying a DNS Resolver

To offer a global DOH service, each edge server needs to run its own caching DNS resolver. A resolver processes the DNS query and caches responses so that users benefit from faster future lookups.

We learned of Knot Resolver by talking to friends in the CDN industry. Knot Resolver is open source software developed by the Czech Republic DNS registry (CZ.NIC). Fun fact, Knot Resolver also powers the Cloudflare 1.1.1.1 public DNS service. Others recommended Unbound or BIND as resolvers. We may yet run those in a mixed network for heterogenous resilience.

It took a few days to tune the Knot Resolver configuration and automate its deployment. This involved remotely upgrading the operating system across all edge servers. A risky proposition involving custom vendor kernels for the ARM servers. With overseas unattended physical machines there is no option to press a reset button or flip a power switch. The only solution was to perform tireless careful testing in staging environments, using Vagrant/Virtualbox or on spare hardware. Eventually we ironed out subtle differences between the various server configurations.

This was much more time consuming and technically challenging than coding Playdoh. Our knowledge and experience continues to grow, as documented in the merge request description. Next time this will be easy.

So how is DNS traffic served by a DOH CDN?

Bootstrapping DOH CDN: Anycast IP vs Geo DNS

Users need to be able to easily configure their DNS settings and connect to a nearby DNS server for low-latency lookups.

Traditional public DNS services make use of an expensive Anycast IP network. Users are routed to one of many edge servers worldwide. They share the same IP address but announce different routes using BGP at Internet Exchanges. ISPs will route users via the shortest path. Sadly this is not easily accessible due to cost and administrative overhead.

They also opt for memorable IP addresses. Google owns 8.8.8.8, Cloudflare owns 1.1.1.1, Quad9 (IBM) owns 9.9.9.9, and so on. Their IPv6 addresses are less human friendly but the principle is the same.

With DOH the DNS resolver address is a familiar URL instead of an IP address. This URL contains a domain name so that the connection may be secured using a signed TLS certificate. E.g.: https://commons.host

So DNS itself is used to direct traffic to a DNS over HTTPS service. Chicken or egg? Not quite.

DOH works by bootstrapping the initial DNS lookup of the resolver's hostname. This DNS lookup is still handled by traditional DNS servers like those of an ISP or a local server. The HTTPS connection is then secured with a signed TLS certificate for that domain. Any tampering by a malicious (or faulty) DNS server at the ISP would simply result in a failed connection attempt. So there is no risk of exposing to the DOH client to tampered responses.

Diagram: DOH bootstrap sequence

Bootstrap procedure:

1. Browser performs DNS lookup for the Commons Host DOH server hostname using a standard, potentially untrusted, DNS server.
2. DNS server responds with the IP address of the nearest Commons Host edge server.
3. Browser establishes a HTTP/2 connection with the edge server. TLS certificates ensure an encrypted and authenticated connection.
4. Subsequent DNS lookups are tunnelled inside the HTTP/2 connection to keep them safe from snooping or tampering by third parties.

* Shown IP address is an example. Actual address is based on location and other performance metrics to determine the optimal edge server for a particular user.

What About Security?

Running a public DNS service is typically fraught with security problems. An open resolver, one that accepts DNS queries from anyone on the Internet, is a convenient traffic-amplifier for DDoS botnets and other malicious actors.

Amplification attacks work by spoofing the source address on a small DNS query, so that the large DNS response gets delivered to an unfortunate target. Attackers use public DNS servers to generate a multiple of their own bandwidth and aim it at a target while hiding themselves as originators. Most people would never want to run such a service, and in fact many ISPs block inbound UDP traffic on port 53 for this reason.

DOH eliminates the spoofing problem. The HTTPS connection requires a secure handshake so traffic can not be spoofed or misdirected. Any responses are always delivered to the correct source, making DOH safe from amplification attacks.

Living on the Edge

Running a public DOH service is much easier than a traditional DNS open resolver. Expect many organisations to offer such services.

One benefit of the Commons Host network is that anyone can sponsor and host an edge server. This brings the CDN edge on-premises, and a great way to run a sub-millisecond latency (i.e. LAN) DOH server. Doing so improves your DNS lookup speed while serving your local community. Get in touch if you are interested.

Using Commons Host DOH

Currently Firefox is the easiest way to use DOH.

Screenshot: Firefox Network Settings for DOH

1. In the Preferences screen, open the Connection Settings dialog.
2. Turn on the checkbox: Enable DNS over HTTPS
3. Enter the URL: https://commons.host

More DOH browser/OS support and bridging solutions will hopefully follow soon. Chrome seems to have a DOH implementation on the way.

One Last Thing: Custom Domain DOH Resolver

Commons Host supports DOH service on custom domains!

Deploying a custom domain on Commons Host is as easy as pointing a CNAME DNS record to commons.host using your domain name management provider.

Commons Host uses Geo DNS to point the domain commons.host to the edge server best suited for any user worldwide. Every edge server has its own public IP address. The edge web servers run the Playdoh middleware which processes DOH requests based on HTTP headers, while regular web requests pass through. The same domain and the same edge server can serve both web and DOH traffic.

Simply deploy a website with a custom domain and use your personal URL as the DOH resolver endpoint. For example: https://www.$yourdomain.com

The same goes for choosing a specific Commons Host edge server as your DOH endpoint. Each one is directly addressable by country code, airport code, and incrementing counter. For example: https://us-lax-1.commons.host connects directly to the Los Angeles edge server.