Network professionals aside, most of us take access to internet for granted. How packets flow from one end to another is so obscure, involving last mile networks, transit providers, public exchanges and even more complex delivery methods. The recent Netflix/Comcast deal and Netflix/Verizon dispute has brought the guts of how these tubes work into the headlines, but it’s wrong to think that only Netflix was impacted.
Beyond Netflix: Widespread Impact
The dispute impacted any company that relied on the Backbone ISPs in question and any communication that was routed through the impacted peering points. It impacted your end users accessing your webpages, or your partners/clients accessing your APIs, and in some cases, your CDN requesting data from your origin servers.
At Catchpoint, we have 116 nodes in the US, actively monitoring from various backbone providers and consumer ISPs. We see such peering problems frequently, usually limited to specific locations and solved by ISPs within a couple of weeks at most. However, we’ve seen this particular issue in the Northeast since spring of 2013. Several major infrastructure companies (CDNs, DNS, etc) were well aware of the issue and adjusted their routes accordingly – but many websites and services are being impacted.
To highlight the issue, we are providing data on Ping Latency captured from our agents in the North East.
Consumer ISPs to Backbone
To assess Consumer ISP/Backbone peering, we measured Ping round trip times from consumer ISPs to Backbone networks. The chart below shows the last 30 days of data by hour-of-day from Verizon Fios locations to NTT and Cogent in Ashburn, Virginia (where the majority of Northeast datacenters are located).
You can clearly see Ping round trips times slowing down during peak times.
Ping round trip time from VZN FIOS (NY, BOSTON, DC) to NTT by Hour of Day
Ping round trip time from VZN FIOS (NY, BOSTON, DC) to Cogent by Hour of Day
Backbone to Backbone – Light in the End of The Tunnel
We performed the same tests between Backbone networks to measure Backbone/Backbone peering. The good news is that in last couple of weeks we are seeing the latency normalize between the Backbone providers.
Taking a look at the last 30 days of data between various Backbone nodes in Ashburn and Verizon Backbone in NY, network volatility is obvious, but at least the ping round trip times have been dropping recently. It seems like Verizon is working on making these peering points betters.
The improvement is more obvious when comparing the ‘Hour of Day’ data for January and the last 30 days.
Ping round trip time from Ashburn (Cogent, Level 3, NTT, Zayo) to Verizon NY Last 30 Days
Ping round trip time from Ashburn (Cogent, Level 3, NTT, Zayo) to Verizon NY By Hour of Day January
Ping round trip time from Ashburn (Cogent, Level 3, NTT, Zayo) to Verizon NY By Hour of Day Last 30 Days
We see improvement in the data from consumer Verizon ISP too, right now limited to DC.
Peering Impacts Us All
We have seen these peering issues since last summer and not just on Cogent, but also Level 3, NTT and Abovenet. Our results show greatest impacts are in the Northeast U.S. From what we have been recording, it is clear that peering is not just a Netflix & Cogent issue, but also an issue consistent across many transit providers.
Unless network providers start working together to improve these connections, companies relying on these networks are going to be affected, not just connecting to their users but every service that requires a connection across the a backbone network.
**On a side note, let’s not all blame these transit providers for your poor streaming experience at home. I look around just in my household and there are 4-5 iPads, Smart TVs…these devices are data hungry and I believe the US last mile infrastructure is to blame. It has to play catch-up with our new lifestyle. We need faster speed, more throughput, and more bandwidth. When you have a 2 lane highway it’s fine for a certain number of cars, but when you add millions of cars, that road needs to be upgraded otherwise you will be in a parking lot.
Mehdi – Catchpoint