Internet Exchange Points (IXPs) are a critical part of a complex network that makes up the Internet. In this article we will explain the role played by IXPs and how you can monitor the public Internet’s performance.
On a very high level, the Internet looks like this:
Don’t know what to think of that image? Don’t worry, we’ll break it down piece by piece in the following sections.
An Autonomous System (AS) represents a set of IP prefixes that belong to a network and are managed by a single organization. Each AS is assigned an Autonomous System Number (ASN), which is unique to the network.
The global Internet consists of tens of thousands of interconnected autonomous systems and a 32-bit (previously 16-bit) ASN is required for any interconnection between two ASs.
The Internet Assigned Numbers Authority (IANA) is responsible for the ASN’s allocation. IANA assigns the ASNs to the five regional Internet registries (RIR). The RIRs, in turn, allocate the ASNs to the organizations in their respective regions.
As mentioned above, the Internet is a network of networks (AS). Most of these AS networks are enterprises, government entities, and universities. These AS networks are connected by Internet Service Providers (ISPs), which are also AS networks (one or more). Each ISP has its own routing policies and topologies that are used inside their AS.
The ISPs transport Internet traffic for other ISPs and for their customers.
Some ISPs are large, with extended reach across the world. Others are smaller — limited to a country or continent. This has lead to a 3-tier ISP model:
Internet Service Providers (ISPs), Content Delivery Networks (CDNs), web-scale, and SaaS companies require exchanging internet traffic to work. This exchange is usually made possible by connecting to internet eXchange Points (IXPs).
An IXP is nothing more than a location spread across one or more sites for redundancy where the members of the IXP are connected in a large Layer 2 network.
Various methods are available to create a single Layer 2 domain (across multiple switches, spread across multiple sites). For example, the process can be as simple as a regular VLAN configured across all switches or it can be something as complex as using Ethernet VPN (EVPN) as a control plane and Virtual Extensible LAN (VXLAN) as a data plane.
For instance, this is a representation of how various entities could be connected to an IXP:
The advantage of having a presence in an IXP is that the members are shortening their paths to other members of the IXP. This reduces latency, improves round-trip time, and lowers costs compared to traffic routed through a transit ISP, as demonstrated in the graphic below
As of January 2021, there are around 840 IXPs, according to PeeringDB. PeeringDB is a database containing networks. It has become the de facto location of Internet peering information.
We noted previously that each AS owns a set of IP prefixes. These prefixes are exchanged between AS using routing protocols. A routing protocol allows the routers to choose the best path (IP prefix) to a destination and to adapt the routing table content in case of topology change events.
The protocol that runs the Internet is Border Gateway Protocol (BGP). The current BGP version is version 4. Thanks to multiprotocol extensions, version 4 can carry information about IP multicast routes and various Layer 3 address families like IPv4, IPv6, VPNv4, and VPNv6.
Each AS uses BGP protocol to exchange IP prefixes with other AS. There are two types of BGP: internal and external. The internal BGP (IBGP) is when the BGP session is configured between two peers within the same AS. The external BGP (EBGP) is when the BGP session is configured between two peers from different AS.
BGP, like any other routing protocol, prefers more specific routes rather than less specific routes. When dealing with prefixes, the longer the subnet mask, the more specific the prefix is. That means a 10.10.10.0/24 prefix is preferred over a 10.10.0.0/16 prefix when the router tries to communicate with 10.10.10.10 host.
When trying to decide which route is the best (in the event two identical routes are received from two BGP neighbours), BGP uses a best-path-selection algorithm that compares attributes and values of those prefixes.
The fact that BGP chooses the more specific prefix has led to massive Internet outages, some by mistake and some intentionally. For example, an enterprise could advertise (by mistake or intentionally) the Office365 prefixes as more specific prefixes than what Microsoft announces. Since every AS prefers the most specific routes possible, in a case like that, all of the traffic intended for Office365 applications would instead be directed to the enterprise.
To avoid such situations, each AS should allow from its BGP peers only the routes that are owned by those peer AS. This method is accomplished widely by using Internet Routing Registries (IRR) which are databases containing associations between AS and prefixes.
A newer mechanism to protect the route origin is Resource Public Key Infrastructure (RPKI), which is a way to cryptographically sign the associations between prefixes and originating AS of those prefixes. RPKI implementation effort is led by CDN providers, web scale providers and some of the major ISPs.
Domain Name System (DNS) translates domain names to IP addresses. All of the Internet runs on IPv4 or IPv6 protocol.
Whenever a website is accessed, the host from where the website is accessed needs to know the IP address of the website. This process is transparent for the user and requires a DNS server to be configured on the host. The DNS server is able to make queries to find out what is the IP address associated with the website.
This type of translation is also called forward DNS.
There is another type of DNS translation called reverse DNS. Reverse DNS occurs when there is an attempt to find the name for an IP address.
A Content Delivery Network (CDN) is a network of distributed servers that minimizes content-loading delays by providing content from multiple locations. This process shortens the distance between each user and content server.
For example, if multiple users from Europe access a website from Asia, there is no point having each user retrieve the information from the origin server in Asia. It is more optimal to host a cached version of the content on a server in Europe. This will improve the latency and user experience. Should the content change on the origin server, the update will be reflected on the cached content.
While the web content is the main type of traffic delivered through CDN, there are other types of traffic that can be delivered as well, like audio streams, video streams, applications, and OS updates.
An IPv4 address is a 32-bit value. This means there can be around 4.2 billion addresses. Some of those IPv4 addresses are reserved for private usage (RFC 1918), fr multicast usage, or research usage.
4.2 billion might sound like an impressive number. However, at this very moment, some RIRs cannot allocate any IPv4 because they’ve depleted their assigned public IPv4 blocks. This problem was inevitable as the internet grew, although various actions were taken to slow down the IPv4 depletion (such Network Address Translation (NAT)).
Ultimately, the decision was made to create a new protocol that accommodates all existing and future needs. Thus, IPv6 was developed. An IPv6 address has a 128-bit value—a vast improvement compared to IPv4.
Theoretically, each host, network device, IoT device, and sensor can be assigned multiple public IPv6 addresses and still be very far from depleting all public IPv6 addresses.
Unfortunately, IPv6 adoption has been slow. Only recently has adoption started to ramp up, with 30% of Alexa Top 1000 websites now reachable over IPv6. Google also predicts around 30% of their users have IPv6 connectivity.
Using these connections (called transport links or underlay), an SD-WAN solution creates an overlay network that is decoupled from the underlay network.
An SD-WAN solution has a centralized control plane. This control plane is responsible for the SD-WAN routes advertisements between branches that are part of the SD-WAN network.
Typically, any SD-WAN solution has a controller that handles SD-WAN routing. The controller is a management system that allows the configuration, monitoring, and troubleshooting of an SD-WAN network and any edge devices responsible for creating the overlay network. In some vendors, these two components (the SD-WAN and the controller) can be on the same device. It also sends traffic over the overlay network according to whatever policies are in place.
Some of the advantages of SD-WAN include:
There are multiple tools you can use to monitor the health or availability of public network resources.
It is important to understand that with any monitoring tool (be it ping, traceroute, or some other solution), the target host must allow the type of traffic used for monitoring or the intermediary network devices to permit this traffic to come through.
Sometimes, a firewall in the middle might drop all the monitoring traffic and the target host will appear as not available. Or other times, the target host will drop the traffic being monitored. Either of these outcomes can lead to false positive alarms. That is why it is important to use your monitoring system to develop a traffic baseline before setting up any types of alerts.
Ping is the most basic tool that can be used to verify the availability of a device that is configured with an IPv4 or IPv6 address. The tool has been available for almost 40 years.
The ping utility works by sending Internet Control Message Protocol (ICMP) echo requests packets towards the target and waiting for ICMP echo replies. The tool records packet loss and any errors. It also provides a summary of the minimum, maximum, and average round-trip time.
Based on the implementation, additional options can be set when sending the ICMP packets. For instance, you can record the size of the packet, how many packets are sent, to not fragment the packets, to limit the number of network hops, how many packets to send, and other more sophisticated options.
Traceroute is a monitoring tool that is used to reveal, in real-time, the path a packet takes to reach a certain target. The output shows the number of intermediary hops, as well as the time it took to reach all intermediary hops and the target. Most commonly, the traceroute tool uses ICMP packets. However, implementations using UDP or TCP packets are available as well.
It is critical to know what kind of packets the tool is using. This knowledge enables you to understand what needs to be allowed through any potential middle firewall or at the target level. The tool sends packets with increasing TTL values for each set of probes sent (by default three packets).
Multiple traceroute commands executed from numerous sources towards the same destination can be used to track the availability of the host or application. This helps the owner of the application know if there is a local problem with specific users or a more general problem with the application itself.
This is part of what is called synthetic monitoring; a generic name applied to applications monitoring solutions. A synthetic monitoring solution is able to connect from hundreds of points to a specific application or website (mimicking real users) to monitor the availability, performance, and health of the application or website.
The purpose of synthetic monitoring can range from competition performance analysis, to third party performance evaluation, to discovering user experience improvements.
The information about BGP AS and the prefixes that belong to each AS is public. For any given prefix, various AS will have a different view of that prefix and how they reach it. Collecting such information from various points can allow BGP monitoring tools to detect when changes in routing have appeared and if those changes caused any service impact.
In addition, the BGP monitoring can detect route hijacking, isolate bottlenecks, and route changes between customer locations and business applications.
DNS monitoring allows network connectivity testing between the authoritative name servers and local recursive servers. Also, DNS monitoring can ensure that the DNS records are not changed mistakenly due to human errors or malicious attacks.
With regards to the malicious attacks, two methods are more common:
CDN monitoring allows to detect any performance issues between users, origin, and edge. A CDN monitoring solution measures the page load times, fetch times, latency, and loss and it can map users to specific CDN edge locations based on the user location.
Internet Exchange Points interconnect the backbone of the Internet. It is a challenging task to monitor such a vast delivery chain, which traverses across multiple ISPs and ASNs. The end results, though, are better performing applications with happy end users. This translates into business outcomes such as more business closed, increased customer loyalty, improved employee productivity, and streamlined business processes.
The solution to isolating performance bottlenecks is to insert probes at every step along the path of your public area network. The probes must be deployed in a global network of nodes to cast as wide and diverse of a reach as possible. Read our guide to Synthetic Monitoring to learn more.
Develop a killer DEM strategy.
Our one-page checklist will help you determine your monitoring strategy and data analysis essentials.
"At Arc XP, we make sure that the platform is scalable, is efficient, and is going to meet the demands of all of our customers, which frees up capacity for engineers in these organizations to really focus on site performance. We've partnered with the best in the business to achieve this. Obviously, Catchpoint is one of them."Learn how Arc XP uses Catchpoint to build a culture of performance and speed