CenturyLink outage affects more than just CenturyLink customers

CenturyLink Outage 911: UPDATE Dec 28, 2018

The CenturyLink outage affected 911 services in Washington state, Missouri, Idaho, and Arizona. People report receiving busy dial tones when trying to call 911.

CenturyLink tweeted about “additional technical problems” around 6:30 a.m. EST.

Residents of these areas also report receiving mobile phone alerts about the 911 service outage. There are still reports of 911 outages although 911 service is back up in many areas.

If your area is affected, call your local emergency number. Find a list of local numbers in the KOMO article.

Original article Dec 27, 2018

If your Internet service wasn’t great today, it might be because one of the biggest Internet Service Providers (ISP) is experiencing a major outage.

To deliver superb digital experiences to your customers, you rely on Internet Service Providers and Backbone providers. This is why at Catchpoint, we invest heavily in our node infrastructure so our clients can monitor performance from as many vantage points as possible.

At 4:30 AM a.m. EST on 28 December 2018—our Real User Monitoring (RUM) data alerted us of a steep dip in the #Pageviews originating from CenturyLink ISP across the United States.

The telecommunications giant, who purchased Level3 in November 2017, is experiencing a nationwide outage. To make matters worse, as a major backbone ISP, the impact was not only felt by CenturyLink customers but spread to any traffic that passed through CenturyLink’s network.

Our Synthetic tests running from the nodes using CenturyLink as an ISP also detected the issue and began reporting failures. The first major failures seem to be reported from the West Coast of the US.

Synthetic tests show delayed webpage response times in milliseconds:

Webpage response times:

The Network Error Logging feature we released this past September allowed us to look at the Network errors for RUM data.

https://w3c.github.io/network-error-logging/

The data above is for a sample website for the last 24 hours. It shows “network errors” after filtering the ISP to include only CenturyLink.

We found that the outage impacted transit and peering between ISPs. We ran traceroute tests showing packet loss, timeout errors, and an increase in round-trip time.

Traceroute showing packet loss/timeout:

Traceroute showing increased round trip time:

DNS experience test showing a timeout:

CenturyLink noted the disruption via Twitter about 5 hours after the outage started. They made special note that customers are unable to generate tickets via their help portal as well.

We still don’t know the root cause of the outage. Stepping back from the details for a moment, this shows the potential butterfly effect of issues over global networks.

An ISP causes a chain reaction that results in site failures and timeouts everywhere. It also highlights the fact that, despite all the precautions that a company can take, sometimes our application and web performance depend on third parties.

Click here to learn more about staying ahead of third-party outages.