NS1: Why DNS Can Be Difficult to Benchmark

This post is an excerpt from an article previously published by NS1.

Benchmarking DNS is not an easy task. As NS1 describes in their article, they use a variety of different testing methods to accomplish this goal, however, each comes with its own set of benefits and drawbacks. The first method uses their latency-based routing engine called Pulsar, and a JavaScript tag in which they embed in web pages across the internet. This also measures performance form the end user’s actual browser to their network.

The second method is using third-party monitoring tools, including Catchpoint, which has proven to be the best method according to NS1.

One of the issues with using some third-party tools is that their “DIY” approach makes it difficult to use for benchmarking. As NS1 put it, well-respected tools cache DNS responses despite their documentation claiming they don’t. This is an important factor when testing popular production domains against test domains that see little or no traffic. The article mentions that NS1 has had to  “prove the hard way (i.e., generating traffic and analyzing traffic via tcpdump) on more than one occasion that Akamai does not offer 1ms DNS resolution globally versus our ~30ms global time.”

The article also explains how using third-party data was useful in benchmarking DNS providers to make the case for purchasing one of them.

When Dnsperf was trying to replace their previous provider with NS1, “Ultimately they agreed that the objective data generated by Catchpoint’s 380 global nodes presented a more accurate picture of our performance and uptime than what they were able to generate on their own. What’s really key is Catchpoint’s ability to measure from eyeball networks and ISPs as opposed to just relying on measurements taken from infrastructure providers and tier 1 carriers. Those measurements can, of course, be useful, but what you really care about is the resolution performance your actual end users get, and Catchpoint is great as it allows us to measure both.”

NS1 is consistently one of the top performing DNS networks when it comes to raw response times, but that’s only part of the story. With properties doing large amounts of DNS traffic, where only a small fraction of their requests reach their name servers because they aren’t served from ISP caches, NS1 provides a significantly faster experience for their end users.

Time to first byte is the best way to think of it: it doesn’t matter how fast NS1 delivers an answer, if they send a single user on Time Warner in Southern California to New York because their Geo-IP database is wrong, every single user on that resolver is going to have an additional 70ms added onto their round trip to your servers.

To read the full article, click here.