The emergency 911 system was down for more than an hour on Monday in towns and cities in 14 US states. The outage led several news outlets to speculate related to the problem MicrosoftOf Blue Web services platform, which at the time was also struggling to spread widely. However, several sources suggest that KrebsOnScurity includes 911 issues, including some type of technical snuff Intrado And Lumen, Two companies that jointly handle 911 calls to a wide swath of the United States.
On Monday, September 28, 911 reported migration to various cities and areas in several states, including Arizona, California, Colorado, Delaware, Florida, Illinois, Indiana, Minnesota, Nevada, North Carolina, North Dakota, Ohio, Pennsylvania, and Washington.
Several news suggested that the outages may be related to ongoing service disruptions at Microsoft. But a spokesperson for the software giant told KrebsOnsecurity, “We have not seen any indication that the multi-state 911 outage was the result of a disturbance in yesterday’s Azure service.”
Inquiries with emergency dispatch centers in several towns and cities affected by the 911 outage point to a different source: Omaha, Neb. Intrado – known as last year West Security Communications – Provider of 911 and emergency communications infrastructure, systems and services to telecommunications companies and public safety agencies throughout the country.
Intrado did not respond to multiple requests for comment. But according to Henderson County, NC officials, who experienced their own 911 failures yesterday, Intrado said the outage was the result of a problem with an unspecified service provider.
“On September 28, 2020, at 4:30 pm MT, our 911 service provider overviewed the internal conditions for its network, resulting in an impact on 911 call delivery,” reads a statement intrado to county officials. “The impact was mitigated, and service was restored and confirmed to be functional by MT at 5:47 am. Our service provider is currently working to determine the root cause.”
The service provider referenced in Intrado’s statement, Lumen, appears to be a communications firm and 911 provider, until recently known Centurylink Inc.. A look at the company’s status page on Monday indicates total or partial service disruptions to several lumen systems, including the private and internal cloud networks and its control systems network.
In a statement provided to KrebsOncity, Lumen blamed the issue on Intrado.
The statement said, “At approximately 4:30 pm MT, some Lumen customers were affected by a vendor partner event affecting 911 services, which were AZ, CO, NC, ND, MN, SD and UT. Is in. ” “Service was restored in less than an hour and at this time all 911 traffic is operating properly. The seller partner is in the process of investigating the incident. “
It cannot be an accident that these two companies are now operating under new names, as this will hardly be the first time 911 access has been interrupted for a significant number of Americans between these two.
In 2019, Intrado / West and CenturyLink agreed to pay $ 575,000 to settle an investigation Federal Communications Commission (FCC) Aug. An outage lasting 65 minutes in 2018. The FCC found that the incident was the result of a West security technician colliding a configuration change in the company’s 911 routing network.
On 6 April 2014, some 11 million people across the United States were disconnected from 911 services for 8 hours due to a “completely preventable” software error tied to Intrado’s system. The incident affected 81 call dispatch centers, deactivating emergency services in all of Washington and parts of North Carolina, South Carolina, Pennsylvania, California, Minnesota and Florida.
According to a 2014 Washington Post story about a subsequent investigation and report released by the FCC, there was a problem with IntraDo’s automated system providing a unique identification code to give each incoming call a suitable identification To which it should respond to appropriate “public safety”, or PSAP.
“On April 9, the software responsible for assigning the code maximized over a pre-determined limit,” Post explained. The counter literally stopped counting at 40 million calls. As a result, the routing system stopped accepting new calls, causing a bottleneck and a series of cascading failures elsewhere in the 911 infrastructure. ”
Comparing the length of the 2014 outage, the FCC found that intrado servers were responsible for classifying and placing service disruptions, classifying them as “low-level” events that were never manualized by a human. Were not marked for review.
The FCC eventually fined Intrado and Centurylink $ 17.4 million for the Multirado 2014 outage. An FCC spokesman declined to comment on the outage on Monday, but said the agency was investigating the incident.
Tags: 911 outage, Azure, CenturyLink, FCC, Federal Communications Commission, Intrado, Microsoft, The Washington Post, West Corporation